[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:14 dist-test-slave-hfc3 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:14 dist-test-slave-hfc3 krb5kdc[399](info): set up 2 sockets
Jun 20 14:24:14 dist-test-slave-hfc3 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:24:15.143486 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:15 dist-test-slave-hfc3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:15.263005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43747
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:15.344674 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:15.344875 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:15.344897 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:15.346035 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:15.346077 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:15.346091 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:15.346104 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:15.346117 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:15.347596 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:15.347889 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:15.348140 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:15.353502 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:15.353653 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:15.353725 412 server_base.cc:1034] running on GCE node
W20260620 14:24:15.353502 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:15.354030 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:15.354388 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:15.355587 412 hybrid_clock.cc:648] HybridClock initialized: now 1781965455355553 us; error 44 us; skew 500 ppm
Jun 20 14:24:15 dist-test-slave-hfc3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:15.362166 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:15.362731 412 webserver.cc:460] Webserver started at http://127.0.98.62:34653/ using document root <none> and password file <none>
I20260620 14:24:15.362959 412 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:15.363027 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:15.363188 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:15.364244 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "c407bfab978f4d0294497145700b8d2f"
format_stamp: "Formatted at 2026-06-20 14:24:15 on dist-test-slave-hfc3"
server_key: "afa8a90bdda0c28339d976cc4603ca0b"
server_key_iv: "d22a5f8c5b8676bfe37f27b62870cf30"
server_key_version: "encryptionkey@0"
I20260620 14:24:15.364630 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "c407bfab978f4d0294497145700b8d2f"
format_stamp: "Formatted at 2026-06-20 14:24:15 on dist-test-slave-hfc3"
server_key: "afa8a90bdda0c28339d976cc4603ca0b"
server_key_iv: "d22a5f8c5b8676bfe37f27b62870cf30"
server_key_version: "encryptionkey@0"
I20260620 14:24:15.366122 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:15.367066 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:15.367326 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:15.367436 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "c407bfab978f4d0294497145700b8d2f"
format_stamp: "Formatted at 2026-06-20 14:24:15 on dist-test-slave-hfc3"
server_key: "afa8a90bdda0c28339d976cc4603ca0b"
server_key_iv: "d22a5f8c5b8676bfe37f27b62870cf30"
server_key_version: "encryptionkey@0"
I20260620 14:24:15.367527 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:15.381084 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:15.383311 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:15.383479 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:15.387672 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43747
I20260620 14:24:15.387717 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43747 every 8 connection(s)
I20260620 14:24:15.388127 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:15.388988 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:15.391450 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f: Bootstrap starting.
I20260620 14:24:15.392061 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:15.392370 480 log.cc:824] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:15.393100 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f: No bootstrap required, opened a new log
I20260620 14:24:15.394753 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c407bfab978f4d0294497145700b8d2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43747 } }
I20260620 14:24:15.394894 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:15.394920 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c407bfab978f4d0294497145700b8d2f, State: Initialized, Role: FOLLOWER
I20260620 14:24:15.395125 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c407bfab978f4d0294497145700b8d2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43747 } }
I20260620 14:24:15.395202 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:15.395251 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:15.395305 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:15.395820 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c407bfab978f4d0294497145700b8d2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43747 } }
I20260620 14:24:15.395949 480 leader_election.cc:302] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c407bfab978f4d0294497145700b8d2f; no voters:
I20260620 14:24:15.396149 480 leader_election.cc:288] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:15.396206 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:15.396397 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [term 1 LEADER]: Becoming Leader. State: Replica: c407bfab978f4d0294497145700b8d2f, State: Running, Role: LEADER
I20260620 14:24:15.396427 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:15.396484 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c407bfab978f4d0294497145700b8d2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43747 } }
I20260620 14:24:15.396925 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c407bfab978f4d0294497145700b8d2f. Latest consensus state: current_term: 1 leader_uuid: "c407bfab978f4d0294497145700b8d2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c407bfab978f4d0294497145700b8d2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43747 } } }
I20260620 14:24:15.397009 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:15.397171 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c407bfab978f4d0294497145700b8d2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c407bfab978f4d0294497145700b8d2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43747 } } }
I20260620 14:24:15.397311 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:15.397280 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:15.398001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 412
I20260620 14:24:15.398113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:15.398137 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:15.398904 392 external_mini_cluster.cc:1428] Setting key 85828321f78ae8a913f35ce66c29e021
I20260620 14:24:15.400214 492 catalog_manager.cc:1347] Generated new cluster ID: 14b4a05c87404011a974199534cb93bb
I20260620 14:24:15.400309 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:24:15 dist-test-slave-hfc3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:15.417215 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:15.400114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.0.98.62:43747)
0620 14:24:15.400283 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:15.400292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:15.400327 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:24:15.402562 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:15.402569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:15.402621 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:15.402744 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:15.404844 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.406225 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.407782 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.408359 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.408447 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:15.412902 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:15.412907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:15.412907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:15.412918 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:15.414953 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:15.415457 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:15.415460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:15.415460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:15.415572 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:15.415877 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:15.415877 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:15.415878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:15.416196 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:15.416325 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:15.416814 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 14:24:15.416965 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260620 14:24:15.417874 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:15.418463 492 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:15.427124 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c407bfab978f4d0294497145700b8d2f: Generated new TSK 0
I20260620 14:24:15.427313 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:15.456548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43747
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:15.538475 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:15.538683 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:15.538715 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:15.539793 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:15.539824 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:15.539850 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:15.541303 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:15.541538 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:15.541822 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:15.546828 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:15.546828 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:15.546837 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:15.547245 506 server_base.cc:1034] running on GCE node
I20260620 14:24:15.547405 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:15.547595 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:15.548740 506 hybrid_clock.cc:648] HybridClock initialized: now 1781965455548741 us; error 39 us; skew 500 ppm
Jun 20 14:24:15 dist-test-slave-hfc3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:15.551373 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:15.551805 506 webserver.cc:460] Webserver started at http://127.0.98.1:37641/ using document root <none> and password file <none>
I20260620 14:24:15.552029 506 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:15.552074 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:15.552220 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:15.553390 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "3dd50459f1764e7aa7ade25e0bafefc3"
format_stamp: "Formatted at 2026-06-20 14:24:15 on dist-test-slave-hfc3"
server_key: "0687a931cbfcd7ad448439dc5552f0e2"
server_key_iv: "125b9fc87cbfce6eec8e8f4c9c822003"
server_key_version: "encryptionkey@0"
I20260620 14:24:15.553771 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3dd50459f1764e7aa7ade25e0bafefc3"
format_stamp: "Formatted at 2026-06-20 14:24:15 on dist-test-slave-hfc3"
server_key: "0687a931cbfcd7ad448439dc5552f0e2"
server_key_iv: "125b9fc87cbfce6eec8e8f4c9c822003"
server_key_version: "encryptionkey@0"
I20260620 14:24:15.555238 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:15.556047 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:15.556322 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:24:15.556403 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "3dd50459f1764e7aa7ade25e0bafefc3"
format_stamp: "Formatted at 2026-06-20 14:24:15 on dist-test-slave-hfc3"
server_key: "0687a931cbfcd7ad448439dc5552f0e2"
server_key_iv: "125b9fc87cbfce6eec8e8f4c9c822003"
server_key_version: "encryptionkey@0"
I20260620 14:24:15.556495 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:15.585038 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:15.587059 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:15.587227 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:15.587486 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:15.587811 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:15.587848 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:15.587872 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:15.587921 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:15.594767 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40347
I20260620 14:24:15.594834 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40347 every 8 connection(s)
I20260620 14:24:15.595201 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:24:15 dist-test-slave-hfc3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:15.601750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 506
I20260620 14:24:15.601864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:15.602054 392 external_mini_cluster.cc:1428] Setting key 2cad831be1d6fd876eae13f67f78dac8
I20260620 14:24:15.605386 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:15.596053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39871 (local address 127.0.98.62:43747)
0620 14:24:15.596097 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:15.596099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:15.596521 (+ 422us) server_negotiation.cc:407] Connection header received
0620 14:24:15.597165 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:15.597166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:15.597179 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:15.597232 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:15.598248 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.598776 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.599730 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.599922 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.599975 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:15.602321 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:15.602323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:15.602323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:15.602355 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:15.603922 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:15.604374 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:15.604376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:15.604376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:15.604442 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:15.604708 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:15.604709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:15.604709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:15.604835 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:15.604907 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:15.605197 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:24:15.605289 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:24:15.605594 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:15.596037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43747 (local address 127.0.98.1:39871)
0620 14:24:15.596247 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:15.596283 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:15.597076 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:15.597288 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:15.597291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:15.597883 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:15.598161 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:15.598167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.598854 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.598854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:15.599655 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:15.599657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.599999 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.600000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:15.600106 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:15.600705 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:15.600713 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:15.602218 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:15.604035 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:15.604043 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:15.604046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:15.604289 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:15.604513 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:15.604514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:15.604514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:15.604639 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:15.604912 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:15.604918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:15.605065 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 14:24:15.605176 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:24:15.605356 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":138,"threads_started":1}
I20260620 14:24:15.606397 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43747
I20260620 14:24:15.606518 635 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:15.606700 635 heartbeater.cc:511] Master 127.0.98.62:43747 requested a full tablet report, sending...
I20260620 14:24:15.607354 444 ts_manager.cc:194] Registered new tserver with Master: 3dd50459f1764e7aa7ade25e0bafefc3 (127.0.98.1:40347)
I20260620 14:24:15.608364 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39871
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:15.629308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43747
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:15.712218 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:15.712471 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:15.712513 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:15.713686 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:15.713765 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:15.713814 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:15.715448 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:15.715718 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:15.715962 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:15.720963 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:15.720963 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:15.721033 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:15.721323 640 server_base.cc:1034] running on GCE node
I20260620 14:24:15.721518 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:15.721807 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:15.722945 640 hybrid_clock.cc:648] HybridClock initialized: now 1781965455722938 us; error 30 us; skew 500 ppm
Jun 20 14:24:15 dist-test-slave-hfc3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:15.725515 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:15.726130 640 webserver.cc:460] Webserver started at http://127.0.98.2:36495/ using document root <none> and password file <none>
I20260620 14:24:15.726325 640 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:15.726420 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:15.726521 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:15.727587 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "42b0ad3d3b884cea85fef48cd8eb5d48"
format_stamp: "Formatted at 2026-06-20 14:24:15 on dist-test-slave-hfc3"
server_key: "0501ddaa5b7e377a05fa4c4a03399e1a"
server_key_iv: "2ca5352b8d38ab1721a017fefeb7d5f4"
server_key_version: "encryptionkey@0"
I20260620 14:24:15.727932 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "42b0ad3d3b884cea85fef48cd8eb5d48"
format_stamp: "Formatted at 2026-06-20 14:24:15 on dist-test-slave-hfc3"
server_key: "0501ddaa5b7e377a05fa4c4a03399e1a"
server_key_iv: "2ca5352b8d38ab1721a017fefeb7d5f4"
server_key_version: "encryptionkey@0"
I20260620 14:24:15.729254 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:15.730019 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:15.730208 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:15.730322 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "42b0ad3d3b884cea85fef48cd8eb5d48"
format_stamp: "Formatted at 2026-06-20 14:24:15 on dist-test-slave-hfc3"
server_key: "0501ddaa5b7e377a05fa4c4a03399e1a"
server_key_iv: "2ca5352b8d38ab1721a017fefeb7d5f4"
server_key_version: "encryptionkey@0"
I20260620 14:24:15.730451 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:15.747956 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:15.750025 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:15.750185 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:15.750509 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:15.750932 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:15.750993 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:15.751034 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:15.751062 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:15.763134 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40845
I20260620 14:24:15.763172 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40845 every 8 connection(s)
I20260620 14:24:15.763545 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:15.764943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 640
I20260620 14:24:15.765028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:15.765223 392 external_mini_cluster.cc:1428] Setting key 2f2bf78071541d502fd066602913b430
Jun 20 14:24:15 dist-test-slave-hfc3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:15.775298 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:15.764489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46069 (local address 127.0.98.62:43747)
0620 14:24:15.764557 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:15.764559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:15.764899 (+ 340us) server_negotiation.cc:407] Connection header received
0620 14:24:15.765702 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:15.765703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:15.765716 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:15.765755 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:15.766925 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.767417 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.768794 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.768987 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.769051 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:15.771826 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:15.771828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:15.771828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:15.771837 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:15.773550 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:15.774091 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:15.774093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:15.774093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:15.774174 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:15.774527 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:15.774529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:15.774529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:15.774738 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:15.774829 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:15.775087 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 14:24:15.775182 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:24:15.775336 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:15.764468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43747 (local address 127.0.98.2:46069)
0620 14:24:15.764694 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:15.764729 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:15.765599 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:15.765829 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:15.765833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:15.766571 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:15.766824 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:15.766830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.767512 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.767513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:15.768698 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:15.768701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.769078 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.769079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:15.769270 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:15.769851 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:15.769861 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:15.771744 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:15.773679 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:15.773686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:15.773688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:15.774002 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:15.774273 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:15.774274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:15.774274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:15.774449 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:15.774827 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:15.774832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:15.774947 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 14:24:15.775019 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:24:15.775145 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":130,"threads_started":1}
I20260620 14:24:15.776043 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43747
I20260620 14:24:15.776193 769 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:15.776422 769 heartbeater.cc:511] Master 127.0.98.62:43747 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:24:15.776818 444 ts_manager.cc:194] Registered new tserver with Master: 42b0ad3d3b884cea85fef48cd8eb5d48 (127.0.98.2:40845)
I20260620 14:24:15.777184 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:15.793007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43747
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:15.892443 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:15.892692 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:15.892731 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:15.893913 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:15.893976 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:15.894022 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:15.895639 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:15.895912 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:15.896142 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:15.902264 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:15.902248 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:15.902237 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:15.902740 774 server_base.cc:1034] running on GCE node
I20260620 14:24:15.902952 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:15.903229 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:15.904426 774 hybrid_clock.cc:648] HybridClock initialized: now 1781965455904407 us; error 32 us; skew 500 ppm
Jun 20 14:24:15 dist-test-slave-hfc3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:15.907183 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:15.907667 774 webserver.cc:460] Webserver started at http://127.0.98.3:44023/ using document root <none> and password file <none>
I20260620 14:24:15.907940 774 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:15.908010 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:15.908166 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:15.909317 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "32ab7eed579a4c2db2a5aa8af6857f6f"
format_stamp: "Formatted at 2026-06-20 14:24:15 on dist-test-slave-hfc3"
server_key: "3c359a13e06a564768edbabc5cd39147"
server_key_iv: "e0c9d5afc4091e0582abb0a126245d1b"
server_key_version: "encryptionkey@0"
I20260620 14:24:15.909767 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "32ab7eed579a4c2db2a5aa8af6857f6f"
format_stamp: "Formatted at 2026-06-20 14:24:15 on dist-test-slave-hfc3"
server_key: "3c359a13e06a564768edbabc5cd39147"
server_key_iv: "e0c9d5afc4091e0582abb0a126245d1b"
server_key_version: "encryptionkey@0"
I20260620 14:24:15.911409 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:15.912199 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:15.912441 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:24:15.912557 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "32ab7eed579a4c2db2a5aa8af6857f6f"
format_stamp: "Formatted at 2026-06-20 14:24:15 on dist-test-slave-hfc3"
server_key: "3c359a13e06a564768edbabc5cd39147"
server_key_iv: "e0c9d5afc4091e0582abb0a126245d1b"
server_key_version: "encryptionkey@0"
I20260620 14:24:15.912633 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:15.923640 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:15.925804 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:15.925987 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:15.926270 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:15.926633 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:15.926693 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:15.926738 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:15.926764 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:15.941329 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44721
I20260620 14:24:15.941395 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44721 every 8 connection(s)
I20260620 14:24:15.941759 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:24:15 dist-test-slave-hfc3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:15.949366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 774
I20260620 14:24:15.949460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:15.949613 392 external_mini_cluster.cc:1428] Setting key 161fb039ca407c6d42c7909676f9bb6d
I20260620 14:24:15.951637 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:15.942767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51049 (local address 127.0.98.62:43747)
0620 14:24:15.942830 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:15.942832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:15.943095 (+ 263us) server_negotiation.cc:407] Connection header received
0620 14:24:15.943712 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:15.943713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:15.943726 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:15.943809 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:15.944783 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.945274 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.946286 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.946530 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.946588 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:15.948674 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:15.948675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:15.948676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:15.948682 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:15.950237 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:15.950677 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:15.950678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:15.950678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:15.950741 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:15.951003 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:15.951004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:15.951005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:15.951119 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:15.951197 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:15.951471 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 14:24:15.951550 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:24:15.951871 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:15.942684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43747 (local address 127.0.98.3:51049)
0620 14:24:15.942897 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:15.942941 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:15.943589 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:15.943866 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:15.943870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:15.944456 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:15.944697 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:15.944703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.945353 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.945354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:15.946194 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:15.946196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.946600 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.946601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:15.946721 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:15.947196 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:15.947204 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:15.948586 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:15.950324 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:15.950330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:15.950362 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:15.950587 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:15.950808 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:15.950809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:15.950810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:15.950920 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:15.951189 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:15.951195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:15.951334 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 14:24:15.951437 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:24:15.951591 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":128,"threads_started":1}
I20260620 14:24:15.952593 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43747
I20260620 14:24:15.952692 903 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:15.952867 903 heartbeater.cc:511] Master 127.0.98.62:43747 requested a full tablet report, sending...
I20260620 14:24:15.953238 444 ts_manager.cc:194] Registered new tserver with Master: 32ab7eed579a4c2db2a5aa8af6857f6f (127.0.98.3:44721)
I20260620 14:24:15.953629 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51049
I20260620 14:24:15.961529 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:24:15 dist-test-slave-hfc3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:24:15 dist-test-slave-hfc3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:24:15.987746 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:15.979733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.0.98.1:40347)
0620 14:24:15.979981 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:15.979985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:15.979998 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:24:15.980025 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:15.980027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:15.980040 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:15.980172 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:15.980688 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.981402 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.982139 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:15.982583 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:15.982643 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:15.984823 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:15.984830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:15.984830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:15.984837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:15.986662 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:15.986996 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:15.986999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:15.986999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:15.987051 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:15.987214 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:15.987215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:15.987215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:15.987346 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:15.987445 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:15.987516 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:24:15.987574 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":168,"threads_started":1}
W20260620 14:24:15.988209 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37538
Jun 20 14:24:15 dist-test-slave-hfc3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:24:16 dist-test-slave-hfc3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:24:16.013897 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:16.006530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.0.98.1:40347)
0620 14:24:16.006586 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:16.006588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:16.006736 (+ 148us) server_negotiation.cc:407] Connection header received
0620 14:24:16.006819 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:16.006820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:16.006834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:16.006901 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:16.007496 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:16.007988 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:16.008766 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:16.008959 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:16.009043 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:16.011456 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:16.011457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:16.011458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:16.011470 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:16.012758 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:16.013101 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:16.013102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:16.013103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:16.013156 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:16.013360 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:16.013361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:16.013362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:16.013457 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:16.013587 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:16.013671 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:24:16.013797 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:24:16.014648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 506
I20260620 14:24:16.019524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 640
I20260620 14:24:16.025409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 774
I20260620 14:24:16.029881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 412
2026-06-20T14:24:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1940 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:16 dist-test-slave-hfc3 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:16 dist-test-slave-hfc3 krb5kdc[917](info): set up 2 sockets
Jun 20 14:24:16 dist-test-slave-hfc3 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 14:24:17.066212 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:17.186231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34043
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:17.266773 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:17.266978 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:17.266999 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:17.268074 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:17.268108 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:17.268122 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:17.268136 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:17.268147 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:17.269510 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:17.269690 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:17.269860 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:17.274580 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:17.274649 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:17.274580 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:17.274890 930 server_base.cc:1034] running on GCE node
I20260620 14:24:17.275156 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:17.275450 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:17.276629 930 hybrid_clock.cc:648] HybridClock initialized: now 1781965457276618 us; error 48 us; skew 500 ppm
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:17.279042 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:17.279475 930 webserver.cc:460] Webserver started at http://127.0.98.62:43111/ using document root <none> and password file <none>
I20260620 14:24:17.279718 930 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:17.279788 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:17.279945 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:17.280977 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "42b25486afaa46f08df441f73f1b02de"
format_stamp: "Formatted at 2026-06-20 14:24:17 on dist-test-slave-hfc3"
server_key: "5ab3238386852d1a314820c5d390cd62"
server_key_iv: "40131b81f438b07bbe193329ee31f0a8"
server_key_version: "encryptionkey@0"
I20260620 14:24:17.281337 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "42b25486afaa46f08df441f73f1b02de"
format_stamp: "Formatted at 2026-06-20 14:24:17 on dist-test-slave-hfc3"
server_key: "5ab3238386852d1a314820c5d390cd62"
server_key_iv: "40131b81f438b07bbe193329ee31f0a8"
server_key_version: "encryptionkey@0"
I20260620 14:24:17.282677 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:17.283351 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:17.283529 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:17.283618 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "42b25486afaa46f08df441f73f1b02de"
format_stamp: "Formatted at 2026-06-20 14:24:17 on dist-test-slave-hfc3"
server_key: "5ab3238386852d1a314820c5d390cd62"
server_key_iv: "40131b81f438b07bbe193329ee31f0a8"
server_key_version: "encryptionkey@0"
I20260620 14:24:17.283696 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:17.291744 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:17.293519 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:17.293622 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:17.297236 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34043
I20260620 14:24:17.297271 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34043 every 8 connection(s)
I20260620 14:24:17.297622 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:17.298552 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:17.300843 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de: Bootstrap starting.
I20260620 14:24:17.300937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 930
I20260620 14:24:17.301064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:17.301221 392 external_mini_cluster.cc:1428] Setting key 709909a9acaf07301b620aeff9bae748
I20260620 14:24:17.301676 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:17.302057 998 log.cc:824] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:17.302814 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de: No bootstrap required, opened a new log
I20260620 14:24:17.305176 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b25486afaa46f08df441f73f1b02de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34043 } }
I20260620 14:24:17.305338 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:17.305372 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b25486afaa46f08df441f73f1b02de, State: Initialized, Role: FOLLOWER
I20260620 14:24:17.305488 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b25486afaa46f08df441f73f1b02de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34043 } }
I20260620 14:24:17.305583 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:17.305616 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:17.305665 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:17.306263 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b25486afaa46f08df441f73f1b02de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34043 } }
I20260620 14:24:17.306398 998 leader_election.cc:302] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42b25486afaa46f08df441f73f1b02de; no voters:
I20260620 14:24:17.306586 998 leader_election.cc:288] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:17.306757 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:17.306883 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:17.306893 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [term 1 LEADER]: Becoming Leader. State: Replica: 42b25486afaa46f08df441f73f1b02de, State: Running, Role: LEADER
I20260620 14:24:17.307009 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b25486afaa46f08df441f73f1b02de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34043 } }
I20260620 14:24:17.307379 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42b25486afaa46f08df441f73f1b02de. Latest consensus state: current_term: 1 leader_uuid: "42b25486afaa46f08df441f73f1b02de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b25486afaa46f08df441f73f1b02de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34043 } } }
I20260620 14:24:17.307449 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:17.307444 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42b25486afaa46f08df441f73f1b02de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b25486afaa46f08df441f73f1b02de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34043 } } }
I20260620 14:24:17.307641 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:17.307761 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:17.308256 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:17.309834 1007 catalog_manager.cc:1347] Generated new cluster ID: bd5b497f13da48bc9fac6689e14599ee
I20260620 14:24:17.309896 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:17.313891 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.301774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.0.98.62:34043)
0620 14:24:17.302008 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.302017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.302043 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:24:17.302565 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.302571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.302622 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.302746 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:17.303548 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.304461 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.305556 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.306054 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.306141 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.309815 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:17.309819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:17.309820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:17.309833 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:17.311852 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.312257 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.312260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.312260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.312362 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.312595 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.312596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.312597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.312886 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:17.313032 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.313539 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.313679 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":139,"threads_started":1}
I20260620 14:24:17.320739 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:17.321288 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:17.331077 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42b25486afaa46f08df441f73f1b02de: Generated new TSK 0
I20260620 14:24:17.331270 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:17.362582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34043
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:17.443138 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:17.443372 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:17.443408 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:17.444550 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:17.444615 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:17.444661 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:17.446095 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34043
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:17.446388 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:17.446620 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:17.451619 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:17.451730 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:17.451725 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:17.452093 1024 server_base.cc:1034] running on GCE node
I20260620 14:24:17.452302 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:17.452528 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:17.453677 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781965457453664 us; error 31 us; skew 500 ppm
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:17.456157 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:17.456537 1024 webserver.cc:460] Webserver started at http://127.0.98.1:34201/ using document root <none> and password file <none>
I20260620 14:24:17.456717 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:17.456758 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:17.456843 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:17.457911 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "d806d10a76294db58fcfd94c6eb30a0e"
format_stamp: "Formatted at 2026-06-20 14:24:17 on dist-test-slave-hfc3"
server_key: "cbbbc1d59424e85a6bfa6c2c7d986294"
server_key_iv: "fd31cf68300e4c1915527508b6aa5956"
server_key_version: "encryptionkey@0"
I20260620 14:24:17.458221 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d806d10a76294db58fcfd94c6eb30a0e"
format_stamp: "Formatted at 2026-06-20 14:24:17 on dist-test-slave-hfc3"
server_key: "cbbbc1d59424e85a6bfa6c2c7d986294"
server_key_iv: "fd31cf68300e4c1915527508b6aa5956"
server_key_version: "encryptionkey@0"
I20260620 14:24:17.459547 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:17.460248 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:17.460420 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:17.460515 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "d806d10a76294db58fcfd94c6eb30a0e"
format_stamp: "Formatted at 2026-06-20 14:24:17 on dist-test-slave-hfc3"
server_key: "cbbbc1d59424e85a6bfa6c2c7d986294"
server_key_iv: "fd31cf68300e4c1915527508b6aa5956"
server_key_version: "encryptionkey@0"
I20260620 14:24:17.460570 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:17.479647 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:17.481792 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:17.481941 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:17.482198 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:17.482570 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:17.482609 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:17.482632 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:17.482681 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:17.488749 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40943
I20260620 14:24:17.488844 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40943 every 8 connection(s)
I20260620 14:24:17.489194 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:17.497571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 1024
I20260620 14:24:17.497665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:17.497833 392 external_mini_cluster.cc:1428] Setting key e191ebffbe0ec27041d0460657b248be
I20260620 14:24:17.498306 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.489898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47711 (local address 127.0.98.62:34043)
0620 14:24:17.489975 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.489978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.490183 (+ 205us) server_negotiation.cc:407] Connection header received
0620 14:24:17.490744 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.490745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.490755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.490810 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:17.491722 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.492214 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.493160 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.493353 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.493404 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.495609 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:17.495610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:17.495611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:17.495617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:17.496886 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.497266 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.497267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.497268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.497319 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.497563 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.497564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.497565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.497716 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:17.497793 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.498173 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.498243 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:24:17.498533 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.489864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34043 (local address 127.0.98.1:47711)
0620 14:24:17.490055 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:17.490094 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:17.490654 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:17.490882 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:17.490886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:17.491449 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:17.491646 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.491651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.492291 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.492291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.493089 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.493090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.493438 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.493439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.493536 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.493990 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:17.493998 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:17.495521 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:17.496990 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:17.496996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:17.496997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:17.497201 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:17.497379 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:17.497380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:17.497380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:17.497475 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:17.497794 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:17.497801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:17.497978 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 14:24:17.498096 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:24:17.498251 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
I20260620 14:24:17.499032 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34043
I20260620 14:24:17.499140 1153 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:17.499328 1153 heartbeater.cc:511] Master 127.0.98.62:34043 requested a full tablet report, sending...
I20260620 14:24:17.499941 962 ts_manager.cc:194] Registered new tserver with Master: d806d10a76294db58fcfd94c6eb30a0e (127.0.98.1:40943)
I20260620 14:24:17.500635 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47711
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:17.521221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34043
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:17.602121 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:17.602404 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:17.602459 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:17.603669 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:17.603744 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:17.603791 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:17.605528 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34043
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:17.605791 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:17.606047 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:17.611461 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:17.611495 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:17.611788 1158 server_base.cc:1034] running on GCE node
W20260620 14:24:17.614487 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:17.614735 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:17.615000 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:17.618301 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781965457618266 us; error 46 us; skew 500 ppm
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:17.621385 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:17.621959 1158 webserver.cc:460] Webserver started at http://127.0.98.2:46713/ using document root <none> and password file <none>
I20260620 14:24:17.622210 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:17.622257 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:17.622403 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:17.623448 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "f27272c2a6f34485a96de5dd404c2a9b"
format_stamp: "Formatted at 2026-06-20 14:24:17 on dist-test-slave-hfc3"
server_key: "41471fa327dc3a62f18f071019370acd"
server_key_iv: "347c14a9f8ae136fe15fbff2c4f809bd"
server_key_version: "encryptionkey@0"
I20260620 14:24:17.623764 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f27272c2a6f34485a96de5dd404c2a9b"
format_stamp: "Formatted at 2026-06-20 14:24:17 on dist-test-slave-hfc3"
server_key: "41471fa327dc3a62f18f071019370acd"
server_key_iv: "347c14a9f8ae136fe15fbff2c4f809bd"
server_key_version: "encryptionkey@0"
I20260620 14:24:17.625206 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:17.625913 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:17.626096 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:17.626163 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "f27272c2a6f34485a96de5dd404c2a9b"
format_stamp: "Formatted at 2026-06-20 14:24:17 on dist-test-slave-hfc3"
server_key: "41471fa327dc3a62f18f071019370acd"
server_key_iv: "347c14a9f8ae136fe15fbff2c4f809bd"
server_key_version: "encryptionkey@0"
I20260620 14:24:17.626212 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:17.643601 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:17.645985 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:17.646142 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:17.646432 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:17.646770 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:17.646804 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:17.646827 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:17.646875 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:17.652958 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33289
I20260620 14:24:17.653076 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33289 every 8 connection(s)
I20260620 14:24:17.653364 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:17.656189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 1158
I20260620 14:24:17.656262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:17.656445 392 external_mini_cluster.cc:1428] Setting key 6b6d35890df61048dba52d3a331d20e7
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:17.663327 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.654145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58353 (local address 127.0.98.62:34043)
0620 14:24:17.654188 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.654190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.654384 (+ 194us) server_negotiation.cc:407] Connection header received
0620 14:24:17.654876 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.654877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.654887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.654944 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:17.655816 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.656364 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.657827 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.658027 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.658082 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.660531 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:17.660532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:17.660532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:17.660539 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:17.661917 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.662328 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.662329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.662330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.662418 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.662688 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.662689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.662689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.662802 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:17.662877 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.663154 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.663235 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:24:17.663573 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.654092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34043 (local address 127.0.98.2:58353)
0620 14:24:17.654247 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:17.654288 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:17.654788 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:17.655000 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:17.655002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:17.655550 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:17.655741 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.655745 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.656452 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.656453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.657740 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.657743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.658109 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.658110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.658251 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.658813 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:17.658830 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:17.660414 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:17.662021 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:17.662027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:17.662029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:17.662249 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:17.662489 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:17.662490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:17.662490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:17.662611 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:17.662868 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:17.662874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:17.663022 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 14:24:17.663134 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:24:17.663290 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260620 14:24:17.664319 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34043
I20260620 14:24:17.664400 1287 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:17.664618 1287 heartbeater.cc:511] Master 127.0.98.62:34043 requested a full tablet report, sending...
I20260620 14:24:17.665015 962 ts_manager.cc:194] Registered new tserver with Master: f27272c2a6f34485a96de5dd404c2a9b (127.0.98.2:33289)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:24:17.665393 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:17.681528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34043
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:17.764892 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:17.765139 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:17.765165 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:17.766268 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:17.766302 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:17.766330 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:17.767867 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34043
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:17.768054 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:17.768232 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:17.773401 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:17.773373 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:17.773332 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:17.773550 1292 server_base.cc:1034] running on GCE node
I20260620 14:24:17.773833 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:17.774076 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:17.775236 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781965457775220 us; error 29 us; skew 500 ppm
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:17.778018 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:17.778494 1292 webserver.cc:460] Webserver started at http://127.0.98.3:43895/ using document root <none> and password file <none>
I20260620 14:24:17.778765 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:17.778837 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:17.778983 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:17.780063 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "9a2e99944670480cbb49da3952107398"
format_stamp: "Formatted at 2026-06-20 14:24:17 on dist-test-slave-hfc3"
server_key: "c0facbe47f0765aa9ece351b2d067d3e"
server_key_iv: "347122c4eb57ea5a8db031c1172b2d51"
server_key_version: "encryptionkey@0"
I20260620 14:24:17.780431 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9a2e99944670480cbb49da3952107398"
format_stamp: "Formatted at 2026-06-20 14:24:17 on dist-test-slave-hfc3"
server_key: "c0facbe47f0765aa9ece351b2d067d3e"
server_key_iv: "347122c4eb57ea5a8db031c1172b2d51"
server_key_version: "encryptionkey@0"
I20260620 14:24:17.781774 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:17.782588 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:17.782749 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:17.782848 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "9a2e99944670480cbb49da3952107398"
format_stamp: "Formatted at 2026-06-20 14:24:17 on dist-test-slave-hfc3"
server_key: "c0facbe47f0765aa9ece351b2d067d3e"
server_key_iv: "347122c4eb57ea5a8db031c1172b2d51"
server_key_version: "encryptionkey@0"
I20260620 14:24:17.782924 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:17.792220 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:17.794134 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:17.794289 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:17.794545 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:17.794836 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:17.794911 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:17.794957 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:17.794986 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:17.801144 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46513
I20260620 14:24:17.801239 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46513 every 8 connection(s)
I20260620 14:24:17.801524 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:17.806406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 1292
I20260620 14:24:17.806520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:17.806680 392 external_mini_cluster.cc:1428] Setting key ead0e1ce552d4f80b4e41f31072c5714
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:17.811803 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.802241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60567 (local address 127.0.98.62:34043)
0620 14:24:17.802307 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.802309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.802547 (+ 238us) server_negotiation.cc:407] Connection header received
0620 14:24:17.803072 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.803073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.803086 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.803143 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:17.804059 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.804556 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.805569 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.805757 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.805841 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.808489 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:17.808490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:17.808491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:17.808497 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:17.810286 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.810806 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.810808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.810808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.810873 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.811134 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.811135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.811135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.811253 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:17.811325 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.811630 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.811712 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:24:17.812000 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.802239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34043 (local address 127.0.98.3:60567)
0620 14:24:17.802421 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:17.802463 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:17.802961 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:17.803211 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:17.803215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:17.803761 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:17.803939 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.803943 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.804652 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.804654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.805458 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.805459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.805850 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.805851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.805982 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.806621 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:17.806631 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:17.808381 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:17.810472 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:17.810480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:17.810482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:17.810713 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:17.810941 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:17.810942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:17.810943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:17.811064 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:17.811335 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:17.811342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:17.811494 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 14:24:17.811608 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 14:24:17.811767 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":118,"threads_started":1}
I20260620 14:24:17.812482 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34043
I20260620 14:24:17.812565 1421 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:17.812755 1421 heartbeater.cc:511] Master 127.0.98.62:34043 requested a full tablet report, sending...
I20260620 14:24:17.813213 962 ts_manager.cc:194] Registered new tserver with Master: 9a2e99944670480cbb49da3952107398 (127.0.98.3:46513)
I20260620 14:24:17.813711 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60567
I20260620 14:24:17.818274 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:24:17.825333 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.819769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.0.98.62:34043)
0620 14:24:17.819818 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.819820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.819903 (+ 83us) server_negotiation.cc:407] Connection header received
0620 14:24:17.819939 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.819940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.819952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.820000 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:17.820527 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.820988 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.821659 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.821828 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.821876 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.823065 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:17.823066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:17.823067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:17.823073 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:17.824440 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.824729 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.824730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.824730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.824776 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.824916 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.824916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.824916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.825070 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:17.825139 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.825223 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.825276 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:24:17.827455 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:17.829116 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:17.835526 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.831348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.0.98.3:46513)
0620 14:24:17.831490 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.831496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.831801 (+ 305us) server_negotiation.cc:407] Connection header received
0620 14:24:17.831871 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.831874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.831890 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.831977 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:17.832232 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.833356 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.834549 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.835095 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.835195 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.835255 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.835412 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.835451 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260620 14:24:17.835551 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.831270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46513 (local address 127.0.0.1:60732)
0620 14:24:17.831679 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:17.831720 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:17.831789 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:17.832063 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:17.832067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:17.832069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:17.832143 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.832145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.833502 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.833503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.834461 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.834464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.835200 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.835201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.835290 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.835303 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:24:17.835397 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:24:17.835479 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":27,"thread_start_us":51,"threads_started":1}
I20260620 14:24:17.835729 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.831502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.0.98.2:33289)
0620 14:24:17.831671 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.831676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.831958 (+ 282us) server_negotiation.cc:407] Connection header received
0620 14:24:17.832004 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.832006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.832020 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.832099 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:17.832303 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.832979 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.834417 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.835177 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.835248 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.835299 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.835590 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.835638 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260620 14:24:17.835765 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.831277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33289 (local address 127.0.0.1:49564)
0620 14:24:17.831886 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:17.831894 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:17.831940 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:17.832177 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:17.832178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:17.832181 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:17.832227 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.832228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.833091 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.833092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.834325 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.834327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.835336 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.835337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.835402 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.835430 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:24:17.835478 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:24:17.835525 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"spinlock_wait_cycles":384,"thread_start_us":68,"threads_started":1}
I20260620 14:24:17.836267 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.831276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40943 (local address 127.0.0.1:57196)
0620 14:24:17.831681 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:17.831720 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:17.831789 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:17.832063 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:17.832067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:17.832071 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:17.832143 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.832146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.833489 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.833490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.834589 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.834591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.835522 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.835523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.836096 (+ 573us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.836102 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:24:17.836162 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:24:17.836202 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":30,"thread_start_us":70,"threads_started":1}
I20260620 14:24:17.836308 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.831501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.0.98.1:40943)
0620 14:24:17.831752 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.831755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.831766 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:24:17.831871 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.831873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.831890 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.832222 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:17.832295 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.833388 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.834666 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.835368 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.835428 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.835490 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.836188 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.836231 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":88,"threads_started":1}
I20260620 14:24:17.836323 1355 tablet_service.cc:1467] Processing CreateTablet for tablet f8d2d30b65b245a3a4e6cccd45bf2fde (DEFAULT_TABLE table=test-table [id=d4fdaaa9a2444e7685c4c65fc53633a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:17.836483 1221 tablet_service.cc:1467] Processing CreateTablet for tablet f8d2d30b65b245a3a4e6cccd45bf2fde (DEFAULT_TABLE table=test-table [id=d4fdaaa9a2444e7685c4c65fc53633a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:17.836740 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d2d30b65b245a3a4e6cccd45bf2fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:17.836829 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d2d30b65b245a3a4e6cccd45bf2fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:17.837167 1087 tablet_service.cc:1467] Processing CreateTablet for tablet f8d2d30b65b245a3a4e6cccd45bf2fde (DEFAULT_TABLE table=test-table [id=d4fdaaa9a2444e7685c4c65fc53633a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:17.837455 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d2d30b65b245a3a4e6cccd45bf2fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:17.839577 1435 tablet_bootstrap.cc:492] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b: Bootstrap starting.
I20260620 14:24:17.840122 1436 tablet_bootstrap.cc:492] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398: Bootstrap starting.
I20260620 14:24:17.840319 1435 tablet_bootstrap.cc:654] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:17.840899 1437 tablet_bootstrap.cc:492] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e: Bootstrap starting.
I20260620 14:24:17.841070 1436 tablet_bootstrap.cc:654] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:17.841099 1435 log.cc:824] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:17.841965 1437 tablet_bootstrap.cc:654] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:17.842062 1436 log.cc:824] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:17.842418 1435 tablet_bootstrap.cc:492] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b: No bootstrap required, opened a new log
I20260620 14:24:17.842505 1435 ts_tablet_manager.cc:1397] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:24:17.842789 1436 tablet_bootstrap.cc:492] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398: No bootstrap required, opened a new log
I20260620 14:24:17.842867 1436 ts_tablet_manager.cc:1397] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:24:17.842914 1437 log.cc:824] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:17.844020 1437 tablet_bootstrap.cc:492] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e: No bootstrap required, opened a new log
I20260620 14:24:17.844086 1437 ts_tablet_manager.cc:1397] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:24:17.845227 1436 raft_consensus.cc:348] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d806d10a76294db58fcfd94c6eb30a0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40943 } } peers { permanent_uuid: "9a2e99944670480cbb49da3952107398" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46513 } } peers { permanent_uuid: "f27272c2a6f34485a96de5dd404c2a9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33289 } }
I20260620 14:24:17.845405 1436 raft_consensus.cc:374] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:17.845461 1436 raft_consensus.cc:729] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2e99944670480cbb49da3952107398, State: Initialized, Role: FOLLOWER
I20260620 14:24:17.845575 1436 consensus_queue.cc:260] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d806d10a76294db58fcfd94c6eb30a0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40943 } } peers { permanent_uuid: "9a2e99944670480cbb49da3952107398" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46513 } } peers { permanent_uuid: "f27272c2a6f34485a96de5dd404c2a9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33289 } }
I20260620 14:24:17.845927 1421 heartbeater.cc:503] Master 127.0.98.62:34043 was elected leader, sending a full tablet report...
I20260620 14:24:17.845937 1436 ts_tablet_manager.cc:1428] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:24:17.845970 1435 raft_consensus.cc:348] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d806d10a76294db58fcfd94c6eb30a0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40943 } } peers { permanent_uuid: "9a2e99944670480cbb49da3952107398" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46513 } } peers { permanent_uuid: "f27272c2a6f34485a96de5dd404c2a9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33289 } }
I20260620 14:24:17.846206 1435 raft_consensus.cc:374] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:17.846279 1435 raft_consensus.cc:729] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27272c2a6f34485a96de5dd404c2a9b, State: Initialized, Role: FOLLOWER
I20260620 14:24:17.846454 1435 consensus_queue.cc:260] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d806d10a76294db58fcfd94c6eb30a0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40943 } } peers { permanent_uuid: "9a2e99944670480cbb49da3952107398" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46513 } } peers { permanent_uuid: "f27272c2a6f34485a96de5dd404c2a9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33289 } }
I20260620 14:24:17.846869 1287 heartbeater.cc:503] Master 127.0.98.62:34043 was elected leader, sending a full tablet report...
I20260620 14:24:17.846902 1435 ts_tablet_manager.cc:1428] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260620 14:24:17.847287 1437 raft_consensus.cc:348] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d806d10a76294db58fcfd94c6eb30a0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40943 } } peers { permanent_uuid: "9a2e99944670480cbb49da3952107398" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46513 } } peers { permanent_uuid: "f27272c2a6f34485a96de5dd404c2a9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33289 } }
I20260620 14:24:17.847527 1437 raft_consensus.cc:374] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:17.847599 1437 raft_consensus.cc:729] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d806d10a76294db58fcfd94c6eb30a0e, State: Initialized, Role: FOLLOWER
I20260620 14:24:17.847728 1437 consensus_queue.cc:260] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d806d10a76294db58fcfd94c6eb30a0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40943 } } peers { permanent_uuid: "9a2e99944670480cbb49da3952107398" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46513 } } peers { permanent_uuid: "f27272c2a6f34485a96de5dd404c2a9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33289 } }
I20260620 14:24:17.848043 1437 ts_tablet_manager.cc:1428] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260620 14:24:17.848160 1153 heartbeater.cc:503] Master 127.0.98.62:34043 was elected leader, sending a full tablet report...
I20260620 14:24:17.886938 1441 raft_consensus.cc:482] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:17.887122 1441 raft_consensus.cc:504] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d806d10a76294db58fcfd94c6eb30a0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40943 } } peers { permanent_uuid: "9a2e99944670480cbb49da3952107398" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46513 } } peers { permanent_uuid: "f27272c2a6f34485a96de5dd404c2a9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33289 } }
I20260620 14:24:17.887647 1441 leader_election.cc:288] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d806d10a76294db58fcfd94c6eb30a0e (127.0.98.1:40943), f27272c2a6f34485a96de5dd404c2a9b (127.0.98.2:33289)
I20260620 14:24:17.891892 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.887799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40943 (local address 127.0.98.3:51235)
0620 14:24:17.887844 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:17.887858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:17.887929 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:17.888426 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:17.888427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:17.888430 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:17.888577 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.888580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.889301 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.889302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.890507 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.890509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.891539 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.891541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.891683 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.891689 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:24:17.891727 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 14:24:17.891781 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 14:24:17.891963 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.887888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56045 (local address 127.0.98.2:33289)
0620 14:24:17.887944 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.887953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.888387 (+ 434us) server_negotiation.cc:407] Connection header received
0620 14:24:17.888472 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.888474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.888486 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.888563 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:17.888808 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.889639 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.891015 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.891573 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.891642 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.891654 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.891848 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.891891 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:24:17.892352 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.887799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33289 (local address 127.0.98.3:56045)
0620 14:24:17.888247 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:17.888258 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:17.888407 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:17.888566 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:17.888567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:17.888568 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:17.888725 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.888727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.889732 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.889733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.890920 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:17.890923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.891684 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.891685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:17.891768 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.891771 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:17.891843 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:24:17.891881 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":191,"thread_start_us":161,"threads_started":1}
I20260620 14:24:17.892390 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.887969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51235 (local address 127.0.98.1:40943)
0620 14:24:17.888220 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.888222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.888239 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:24:17.888269 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.888270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.888304 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.888388 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:17.888683 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.889214 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.890603 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.891426 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.892215 (+ 789us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.892229 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.892249 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.892299 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260620 14:24:17.892931 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d2d30b65b245a3a4e6cccd45bf2fde" candidate_uuid: "9a2e99944670480cbb49da3952107398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d806d10a76294db58fcfd94c6eb30a0e" is_pre_election: true
I20260620 14:24:17.892930 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d2d30b65b245a3a4e6cccd45bf2fde" candidate_uuid: "9a2e99944670480cbb49da3952107398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f27272c2a6f34485a96de5dd404c2a9b" is_pre_election: true
I20260620 14:24:17.893095 1241 raft_consensus.cc:2454] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a2e99944670480cbb49da3952107398 in term 0.
I20260620 14:24:17.893100 1107 raft_consensus.cc:2454] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a2e99944670480cbb49da3952107398 in term 0.
I20260620 14:24:17.893486 1309 leader_election.cc:302] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a2e99944670480cbb49da3952107398, f27272c2a6f34485a96de5dd404c2a9b; no voters:
I20260620 14:24:17.893648 1441 raft_consensus.cc:2781] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:17.893716 1441 raft_consensus.cc:482] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:17.893751 1441 raft_consensus.cc:3037] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:17.894759 1441 raft_consensus.cc:504] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d806d10a76294db58fcfd94c6eb30a0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40943 } } peers { permanent_uuid: "9a2e99944670480cbb49da3952107398" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46513 } } peers { permanent_uuid: "f27272c2a6f34485a96de5dd404c2a9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33289 } }
I20260620 14:24:17.894919 1441 leader_election.cc:288] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [CANDIDATE]: Term 1 election: Requested vote from peers d806d10a76294db58fcfd94c6eb30a0e (127.0.98.1:40943), f27272c2a6f34485a96de5dd404c2a9b (127.0.98.2:33289)
I20260620 14:24:17.895093 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d2d30b65b245a3a4e6cccd45bf2fde" candidate_uuid: "9a2e99944670480cbb49da3952107398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f27272c2a6f34485a96de5dd404c2a9b"
I20260620 14:24:17.895161 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d2d30b65b245a3a4e6cccd45bf2fde" candidate_uuid: "9a2e99944670480cbb49da3952107398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d806d10a76294db58fcfd94c6eb30a0e"
I20260620 14:24:17.895190 1241 raft_consensus.cc:3037] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:17.895253 1107 raft_consensus.cc:3037] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:17.896142 1241 raft_consensus.cc:2454] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2e99944670480cbb49da3952107398 in term 1.
I20260620 14:24:17.896309 1309 leader_election.cc:302] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a2e99944670480cbb49da3952107398, f27272c2a6f34485a96de5dd404c2a9b; no voters:
I20260620 14:24:17.896391 1441 raft_consensus.cc:2781] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:17.896589 1441 raft_consensus.cc:686] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [term 1 LEADER]: Becoming Leader. State: Replica: 9a2e99944670480cbb49da3952107398, State: Running, Role: LEADER
I20260620 14:24:17.896694 1441 consensus_queue.cc:237] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d806d10a76294db58fcfd94c6eb30a0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40943 } } peers { permanent_uuid: "9a2e99944670480cbb49da3952107398" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46513 } } peers { permanent_uuid: "f27272c2a6f34485a96de5dd404c2a9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33289 } }
I20260620 14:24:17.896735 1107 raft_consensus.cc:2454] T f8d2d30b65b245a3a4e6cccd45bf2fde P d806d10a76294db58fcfd94c6eb30a0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2e99944670480cbb49da3952107398 in term 1.
I20260620 14:24:17.897569 962 catalog_manager.cc:5495] T f8d2d30b65b245a3a4e6cccd45bf2fde P 9a2e99944670480cbb49da3952107398 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a2e99944670480cbb49da3952107398 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9a2e99944670480cbb49da3952107398" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d806d10a76294db58fcfd94c6eb30a0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2e99944670480cbb49da3952107398" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f27272c2a6f34485a96de5dd404c2a9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33289 } health_report { overall_health: UNKNOWN } } }
W20260620 14:24:17.903867 1288 tablet.cc:2369] T f8d2d30b65b245a3a4e6cccd45bf2fde P f27272c2a6f34485a96de5dd404c2a9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:24:17.917582 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.907570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.0.98.1:40943)
0620 14:24:17.907629 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.907631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.907864 (+ 233us) server_negotiation.cc:407] Connection header received
0620 14:24:17.907939 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.907940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.907954 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.908053 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:17.908774 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.909624 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.910373 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.910721 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.910800 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.913902 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:17.913914 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:17.913914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:17.913923 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:17.916331 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.916677 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.916681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.916681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.916750 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.916928 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.916929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.916929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.917117 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:17.917289 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.917332 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.917429 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:24:17.946543 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.938326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.0.98.1:40943)
0620 14:24:17.938409 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.938411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.938574 (+ 163us) server_negotiation.cc:407] Connection header received
0620 14:24:17.938607 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.938608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.938621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.938697 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:17.939217 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.940082 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.940787 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.941147 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.941223 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.943738 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:17.943740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:17.943740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:17.943748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:17.945396 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.945707 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.945708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.945709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.945770 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.945936 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.945937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.945937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.946058 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:17.946161 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.946271 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.946399 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:24:17 dist-test-slave-hfc3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:24:17.972951 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:17.965816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.0.98.1:40943)
0620 14:24:17.965906 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:17.965908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:17.966006 (+ 98us) server_negotiation.cc:407] Connection header received
0620 14:24:17.966051 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:17.966052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:17.966062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:17.966108 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:17.966680 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.967180 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.967853 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:17.968055 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:17.968107 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:17.970477 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:17.970478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:17.970478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:17.970485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:17.971983 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.972327 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.972328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.972328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.972382 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:17.972563 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:17.972563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:17.972563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:17.972652 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:17.972727 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:17.972827 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:24:17.972884 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 14:24:17.973563 1067 tablet_service.cc:2908] Rejecting scan request for tablet f8d2d30b65b245a3a4e6cccd45bf2fde: Uninitialized: clean time has not yet been initialized
I20260620 14:24:17.975080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 1024
I20260620 14:24:17.980139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 1158
I20260620 14:24:17.985427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 1292
I20260620 14:24:17.989761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 930
2026-06-20T14:24:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1961 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:18 dist-test-slave-hfc3 krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:18 dist-test-slave-hfc3 krb5kdc[1459](info): set up 2 sockets
Jun 20 14:24:18 dist-test-slave-hfc3 krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260620 14:24:19.030030 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:19 dist-test-slave-hfc3 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:19.148751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46589
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:19.229434 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:19.229627 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:19.229650 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:19.230795 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:19.230835 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:24:19.230849 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:19.230861 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:19.230875 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:19.230886 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:19.232311 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46589
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:19.232488 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:19.232651 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:19.237428 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:19.237498 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:19.237403 1472 server_base.cc:1034] running on GCE node
W20260620 14:24:19.237403 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:19.237926 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:19.238219 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:19.239395 1472 hybrid_clock.cc:648] HybridClock initialized: now 1781965459239380 us; error 53 us; skew 500 ppm
Jun 20 14:24:19 dist-test-slave-hfc3 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:19.241914 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:19.242327 1472 webserver.cc:460] Webserver started at http://127.0.98.62:44099/ using document root <none> and password file <none>
I20260620 14:24:19.242619 1472 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:19.242697 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:19.242851 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:19.243938 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "0d9de597d2064ca5a50b470f19484a6a"
format_stamp: "Formatted at 2026-06-20 14:24:19 on dist-test-slave-hfc3"
server_key: "36e6f76827e25a2b615079d0426fec0f"
server_key_iv: "c8ef98c78f79d33a47ff21e7b5a88e75"
server_key_version: "encryptionkey@0"
I20260620 14:24:19.244292 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d9de597d2064ca5a50b470f19484a6a"
format_stamp: "Formatted at 2026-06-20 14:24:19 on dist-test-slave-hfc3"
server_key: "36e6f76827e25a2b615079d0426fec0f"
server_key_iv: "c8ef98c78f79d33a47ff21e7b5a88e75"
server_key_version: "encryptionkey@0"
I20260620 14:24:19.245579 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:19.246232 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:19.246430 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:19.246527 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "0d9de597d2064ca5a50b470f19484a6a"
format_stamp: "Formatted at 2026-06-20 14:24:19 on dist-test-slave-hfc3"
server_key: "36e6f76827e25a2b615079d0426fec0f"
server_key_iv: "c8ef98c78f79d33a47ff21e7b5a88e75"
server_key_version: "encryptionkey@0"
I20260620 14:24:19.246606 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:19.258383 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:19.260740 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:19.260903 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:19.265475 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46589
I20260620 14:24:19.265548 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46589 every 8 connection(s)
I20260620 14:24:19.265879 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:19.266619 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:19.268750 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a: Bootstrap starting.
I20260620 14:24:19.269292 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:19.269527 1550 log.cc:824] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:19.270129 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a: No bootstrap required, opened a new log
I20260620 14:24:19.272168 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9de597d2064ca5a50b470f19484a6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46589 } }
I20260620 14:24:19.272300 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:19.272321 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9de597d2064ca5a50b470f19484a6a, State: Initialized, Role: FOLLOWER
I20260620 14:24:19.272451 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9de597d2064ca5a50b470f19484a6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46589 } }
I20260620 14:24:19.272532 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:19.272554 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:19.272583 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:19.273049 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9de597d2064ca5a50b470f19484a6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46589 } }
I20260620 14:24:19.273129 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d9de597d2064ca5a50b470f19484a6a; no voters:
I20260620 14:24:19.273269 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:19.273310 1553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:19.273430 1553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [term 1 LEADER]: Becoming Leader. State: Replica: 0d9de597d2064ca5a50b470f19484a6a, State: Running, Role: LEADER
I20260620 14:24:19.273530 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:19.273509 1553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9de597d2064ca5a50b470f19484a6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46589 } }
I20260620 14:24:19.273702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 1472
I20260620 14:24:19.273800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:19.273886 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d9de597d2064ca5a50b470f19484a6a. Latest consensus state: current_term: 1 leader_uuid: "0d9de597d2064ca5a50b470f19484a6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9de597d2064ca5a50b470f19484a6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46589 } } }
I20260620 14:24:19.273976 392 external_mini_cluster.cc:1428] Setting key 1cccdd420dc870014b7a53fa6845c625
I20260620 14:24:19.274047 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:19.275034 1554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d9de597d2064ca5a50b470f19484a6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9de597d2064ca5a50b470f19484a6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46589 } } }
I20260620 14:24:19.275135 1554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a [sys.catalog]: This master's current role is: LEADER
W20260620 14:24:19.275449 1571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:19.275524 1571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:19.275588 1562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:19.275758 1562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:19.277273 1562 catalog_manager.cc:1347] Generated new cluster ID: 2c90e89277c049ccb2ea702c8d2bae24
I20260620 14:24:19.277328 1562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:24:19 dist-test-slave-hfc3 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:19.285563 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.274651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.0.98.62:46589)
0620 14:24:19.274793 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.274800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.274851 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:24:19.275061 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.275069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.275132 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.275326 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:19.276095 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.277356 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.278498 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.279145 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.279266 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.281955 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:19.281960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:19.281960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:19.281972 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:19.283972 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.284359 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.284362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.284363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.284473 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.284672 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.284673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.284673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.284991 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:19.285150 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.285219 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.285354 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":81,"threads_started":1}
I20260620 14:24:19.290160 1562 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:19.290913 1562 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:19.296121 1562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a: Generated new TSK 0
I20260620 14:24:19.296299 1562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:19.325914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46589
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:19.406128 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:19.406354 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:19.406436 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:19.407589 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:19.407629 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:19.407649 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:24:19.407696 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:19.409214 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:19.409440 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:19.409690 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:19.414868 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:19.414841 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:19.414849 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:19.415069 1576 server_base.cc:1034] running on GCE node
I20260620 14:24:19.415366 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:19.415604 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:19.416774 1576 hybrid_clock.cc:648] HybridClock initialized: now 1781965459416752 us; error 33 us; skew 500 ppm
Jun 20 14:24:19 dist-test-slave-hfc3 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:19.419409 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:19.419891 1576 webserver.cc:460] Webserver started at http://127.0.98.1:35563/ using document root <none> and password file <none>
I20260620 14:24:19.420158 1576 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:19.420214 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:19.420348 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:19.421437 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "be0e6ac155b7470eb6e1e18f46ac2822"
format_stamp: "Formatted at 2026-06-20 14:24:19 on dist-test-slave-hfc3"
server_key: "ecb943f65be638158bf0f349339d0ff3"
server_key_iv: "ec6537ca765f83ce82e2164b86567a79"
server_key_version: "encryptionkey@0"
I20260620 14:24:19.421828 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "be0e6ac155b7470eb6e1e18f46ac2822"
format_stamp: "Formatted at 2026-06-20 14:24:19 on dist-test-slave-hfc3"
server_key: "ecb943f65be638158bf0f349339d0ff3"
server_key_iv: "ec6537ca765f83ce82e2164b86567a79"
server_key_version: "encryptionkey@0"
I20260620 14:24:19.423326 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:19.424142 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:19.424335 1576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:24:19.424450 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "be0e6ac155b7470eb6e1e18f46ac2822"
format_stamp: "Formatted at 2026-06-20 14:24:19 on dist-test-slave-hfc3"
server_key: "ecb943f65be638158bf0f349339d0ff3"
server_key_iv: "ec6537ca765f83ce82e2164b86567a79"
server_key_version: "encryptionkey@0"
I20260620 14:24:19.424525 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:19.435724 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:19.437631 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:19.437790 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:19.438418 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:19.438476 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:19.438520 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:19.438547 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:24:19 dist-test-slave-hfc3 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:19.445888 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37375
I20260620 14:24:19.445972 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37375 every 8 connection(s)
I20260620 14:24:19.446310 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:19.448658 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.438785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33705 (local address 127.0.98.62:46589)
0620 14:24:19.438860 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.438862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.439102 (+ 240us) server_negotiation.cc:407] Connection header received
0620 14:24:19.439989 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.439991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.440010 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.440128 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:19.441059 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.441597 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.442775 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.442995 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.443072 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.445683 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:19.445685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:19.445686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:19.445694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:19.447394 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.447812 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.447813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.447814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.447874 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.448133 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.448134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.448135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.448249 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:19.448299 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.448503 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.448573 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:24:19.448740 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.438794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46589 (local address 127.0.98.1:33705)
0620 14:24:19.438947 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.438994 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.439819 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.440213 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.440218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.440630 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:19.440948 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.440955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.441708 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.441709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.442677 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.442679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.443113 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.443114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.443223 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.443738 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:19.443746 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:19.445579 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:19.447469 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:19.447474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:19.447475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:19.447685 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:19.447981 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:19.447982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:19.447982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:19.448075 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:19.448296 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:19.448299 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:19.448401 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.448475 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.448586 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260620 14:24:19.449321 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46589
I20260620 14:24:19.449393 1707 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:19.449565 1707 heartbeater.cc:511] Master 127.0.98.62:46589 requested a full tablet report, sending...
I20260620 14:24:19.450140 1503 ts_manager.cc:194] Registered new tserver with Master: be0e6ac155b7470eb6e1e18f46ac2822 (127.0.98.1:37375)
I20260620 14:24:19.450858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 1576
I20260620 14:24:19.450966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:19.451088 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33705
I20260620 14:24:19.451149 392 external_mini_cluster.cc:1428] Setting key c69369dc71cc123fa1dad96319b725d9
I20260620 14:24:19.457958 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.450505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59205 (local address 127.0.98.62:46589)
0620 14:24:19.450541 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.450543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.450612 (+ 69us) server_negotiation.cc:407] Connection header received
0620 14:24:19.450711 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.450711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.450720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.450778 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:19.451153 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.451913 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.453969 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.454328 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.454431 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.455426 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:19.455427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:19.455427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:19.455434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:19.456843 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.457206 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.457207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.457207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.457259 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.457461 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.457462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.457462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.457570 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:19.457644 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.457797 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.457871 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:24:19.457974 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.450440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46589 (local address 127.0.98.1:59205)
0620 14:24:19.450555 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.450565 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.450623 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.450785 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.450786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.450999 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:19.451071 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.451073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.453368 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.453369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.453881 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.453882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.454431 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.454432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.454513 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.455009 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:19.455014 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:19.455316 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:19.456944 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:19.456946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:19.456946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:19.457115 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:19.457332 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:19.457333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:19.457334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:19.457390 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:19.457636 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:19.457637 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:19.457679 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.457772 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.457871 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:19.477941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46589
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:19.560264 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:19.560472 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:19.560497 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:19.561589 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:19.561625 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:19.561651 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:24:19.561672 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:19.563122 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:19.563301 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:19.563472 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:19.568293 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:19.568346 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:19.568490 1716 server_base.cc:1034] running on GCE node
W20260620 14:24:19.568506 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:19.568742 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:19.569003 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:19.570173 1716 hybrid_clock.cc:648] HybridClock initialized: now 1781965459570157 us; error 27 us; skew 500 ppm
Jun 20 14:24:19 dist-test-slave-hfc3 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:19.572782 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:19.573290 1716 webserver.cc:460] Webserver started at http://127.0.98.2:42327/ using document root <none> and password file <none>
I20260620 14:24:19.573537 1716 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:19.573612 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:19.573764 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:19.575224 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "826d6a469821435bb5223b16b7d0b973"
format_stamp: "Formatted at 2026-06-20 14:24:19 on dist-test-slave-hfc3"
server_key: "ddf69a523d1a927cc45e2f9c61b0adfd"
server_key_iv: "a1d1ec9fe1fed49be78da4fe44294f6e"
server_key_version: "encryptionkey@0"
I20260620 14:24:19.575598 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "826d6a469821435bb5223b16b7d0b973"
format_stamp: "Formatted at 2026-06-20 14:24:19 on dist-test-slave-hfc3"
server_key: "ddf69a523d1a927cc45e2f9c61b0adfd"
server_key_iv: "a1d1ec9fe1fed49be78da4fe44294f6e"
server_key_version: "encryptionkey@0"
I20260620 14:24:19.577031 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:24:19.577796 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:19.577968 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:19.578068 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "826d6a469821435bb5223b16b7d0b973"
format_stamp: "Formatted at 2026-06-20 14:24:19 on dist-test-slave-hfc3"
server_key: "ddf69a523d1a927cc45e2f9c61b0adfd"
server_key_iv: "a1d1ec9fe1fed49be78da4fe44294f6e"
server_key_version: "encryptionkey@0"
I20260620 14:24:19.578184 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:19.590056 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:19.592504 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:19.592675 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:19.593192 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:19.593258 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:19.593303 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:19.593335 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:19.600256 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40019
I20260620 14:24:19.600284 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40019 every 8 connection(s)
Jun 20 14:24:19 dist-test-slave-hfc3 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:19.600677 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:19.603116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 1716
I20260620 14:24:19.603232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:19.603410 392 external_mini_cluster.cc:1428] Setting key f7dcb0781730b856ee7405b64b9a87d7
I20260620 14:24:19.604033 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.593740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35317 (local address 127.0.98.62:46589)
0620 14:24:19.593815 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.593819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.594028 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:24:19.594683 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.594684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.594697 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.594761 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:19.595418 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.596247 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.597881 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.598213 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.598281 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.601157 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:19.601158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:19.601158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:19.601166 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:19.602515 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.602937 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.602938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.602939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.603008 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.603304 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.603305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.603305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.603437 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:19.603519 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.603853 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.603942 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:24:19.604223 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.593710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46589 (local address 127.0.98.2:35317)
0620 14:24:19.593892 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.593931 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.594568 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.594833 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.594836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.595135 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:19.595317 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.595321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.596364 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.596365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.597793 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.597796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.598292 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.598293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.598458 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.599013 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:19.599022 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:19.601036 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:19.602609 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:19.602616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:19.602619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:19.602848 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:19.603090 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:19.603091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:19.603091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:19.603198 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:19.603509 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:19.603516 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:19.603693 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.603830 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.603994 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260620 14:24:19.604717 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46589
I20260620 14:24:19.604806 1847 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:19.604993 1847 heartbeater.cc:511] Master 127.0.98.62:46589 requested a full tablet report, sending...
I20260620 14:24:19.605382 1504 ts_manager.cc:194] Registered new tserver with Master: 826d6a469821435bb5223b16b7d0b973 (127.0.98.2:40019)
I20260620 14:24:19.605907 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35317
I20260620 14:24:19.612255 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.605825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48135 (local address 127.0.98.62:46589)
0620 14:24:19.605864 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.605866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.605975 (+ 109us) server_negotiation.cc:407] Connection header received
0620 14:24:19.606035 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.606036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.606049 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.606118 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:19.606584 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.607166 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.607851 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.608242 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.608322 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.609261 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:19.609263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:19.609263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:19.609271 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:19.610896 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.611327 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.611328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.611329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.611387 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.611600 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.611600 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.611601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.611791 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:19.611838 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.612053 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.612143 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:24:19.612629 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.605779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46589 (local address 127.0.98.2:48135)
0620 14:24:19.605897 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.605908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.605971 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.606164 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.606165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.606429 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:19.606497 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.606500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.607265 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.607266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.607767 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.607768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.608338 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.608339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.608403 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.608867 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:19.608871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:19.609155 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:19.610997 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:19.610998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:19.610999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:19.611235 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:19.611465 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:19.611465 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:19.611466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:19.611527 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:19.611890 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:19.611891 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:19.611936 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.612389 (+ 453us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.612496 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:19.630474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46589
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:19.729219 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:19.729447 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:19.729487 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:19.730696 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:19.730758 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:19.730789 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:24:19.730844 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:19.732340 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:19.732612 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:19.732841 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:19.738005 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:19.738000 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:19.738011 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:19.738030 1856 server_base.cc:1034] running on GCE node
I20260620 14:24:19.738469 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:19.738718 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:19.739903 1856 hybrid_clock.cc:648] HybridClock initialized: now 1781965459739886 us; error 30 us; skew 500 ppm
Jun 20 14:24:19 dist-test-slave-hfc3 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:19.742540 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:19.742987 1856 webserver.cc:460] Webserver started at http://127.0.98.3:38843/ using document root <none> and password file <none>
I20260620 14:24:19.743216 1856 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:19.743290 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:19.743425 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:19.744541 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "593713a3f89e466e978afd3f8afa5e7d"
format_stamp: "Formatted at 2026-06-20 14:24:19 on dist-test-slave-hfc3"
server_key: "de3a6f96f0f01ac661488684b95401dc"
server_key_iv: "f045bfd51dbb3eda2feda3326d839519"
server_key_version: "encryptionkey@0"
I20260620 14:24:19.745087 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "593713a3f89e466e978afd3f8afa5e7d"
format_stamp: "Formatted at 2026-06-20 14:24:19 on dist-test-slave-hfc3"
server_key: "de3a6f96f0f01ac661488684b95401dc"
server_key_iv: "f045bfd51dbb3eda2feda3326d839519"
server_key_version: "encryptionkey@0"
I20260620 14:24:19.746647 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:19.747515 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:19.747749 1856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:19.747864 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "593713a3f89e466e978afd3f8afa5e7d"
format_stamp: "Formatted at 2026-06-20 14:24:19 on dist-test-slave-hfc3"
server_key: "de3a6f96f0f01ac661488684b95401dc"
server_key_iv: "f045bfd51dbb3eda2feda3326d839519"
server_key_version: "encryptionkey@0"
I20260620 14:24:19.747951 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:19.757854 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:19.760002 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:19.760166 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:19.760689 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:19.760757 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:19.760803 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:19.760831 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:19.767925 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42381
I20260620 14:24:19.768015 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42381 every 8 connection(s)
Jun 20 14:24:19 dist-test-slave-hfc3 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:19.768334 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:19.771996 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.761244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50111 (local address 127.0.98.62:46589)
0620 14:24:19.761323 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.761326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.761488 (+ 162us) server_negotiation.cc:407] Connection header received
0620 14:24:19.762134 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.762135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.762150 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.762231 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:19.763117 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.763968 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.765563 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.765899 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.765975 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.768790 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:19.768791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:19.768792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:19.768799 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:19.770490 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.770992 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.770993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.770993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.771045 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.771320 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.771321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.771321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.771425 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:19.771504 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.771865 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.771931 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:24:19.772344 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.761166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46589 (local address 127.0.98.3:50111)
0620 14:24:19.761372 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.761412 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.761995 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.762329 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.762333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.762709 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:19.763011 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.763018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.764069 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.764070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.765450 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.765454 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.765996 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.765996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.766094 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.766667 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:19.766686 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:19.768696 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:19.770568 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:19.770576 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:19.770578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:19.770912 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:19.771106 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:19.771107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:19.771108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:19.771258 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:19.771515 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:19.771523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:19.771706 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.771845 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.772029 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":134,"threads_started":1}
I20260620 14:24:19.772886 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46589
I20260620 14:24:19.772974 1987 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:19.773168 1987 heartbeater.cc:511] Master 127.0.98.62:46589 requested a full tablet report, sending...
I20260620 14:24:19.773671 1503 ts_manager.cc:194] Registered new tserver with Master: 593713a3f89e466e978afd3f8afa5e7d (127.0.98.3:42381)
I20260620 14:24:19.774210 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50111
I20260620 14:24:19.775417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 1856
I20260620 14:24:19.775491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:19.775679 392 external_mini_cluster.cc:1428] Setting key f41045bcdada30ec4b62acae937e2bf6
I20260620 14:24:19.776549 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:24:19.782557 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.774060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58843 (local address 127.0.98.62:46589)
0620 14:24:19.774098 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.774100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.774481 (+ 381us) server_negotiation.cc:407] Connection header received
0620 14:24:19.774729 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.774730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.774739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.774788 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:19.775140 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.775826 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.776718 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.776922 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.777003 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.779303 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:19.779304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:19.779304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:19.779312 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:19.781231 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.781555 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.781556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.781557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.781619 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.781806 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.781807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.781807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.781972 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:19.782098 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.782372 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.782454 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:24:19.782596 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.773987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46589 (local address 127.0.98.3:58843)
0620 14:24:19.774421 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.774432 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.774637 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.774791 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.774792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.774970 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:19.775018 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.775020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.775939 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.775941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.776654 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.776657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.777004 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.777005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.777073 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.778780 (+ 1707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:19.778786 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:19.779202 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:19.781311 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:19.781312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:19.781312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:19.781472 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:19.781687 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:19.781688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:19.781688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:19.781743 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:19.782193 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:19.782194 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:19.782235 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.782302 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.782456 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":75,"threads_started":1}
Jun 20 14:24:19 dist-test-slave-hfc3 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:24:19 dist-test-slave-hfc3 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:19.811256 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.803427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.0.98.62:46589)
0620 14:24:19.803476 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.803478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.803487 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:24:19.803515 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.803516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.803532 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.803614 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:19.804297 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.804856 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.806020 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.806232 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.806305 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.808790 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:19.808792 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:19.808792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:19.808800 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:19.810324 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.810675 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.810676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.810676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.810727 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:19.810887 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:19.810888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:19.810888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:19.810977 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:19.811043 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.811125 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.811191 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:24:19.812927 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:19.814478 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:19.820992 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.816800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42381 (local address 127.0.0.1:49674)
0620 14:24:19.816956 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.816968 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.817018 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.817667 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.817668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.817670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:19.817720 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.817722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.818936 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.818937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.819891 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.819893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.820740 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.820740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.820796 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.820807 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.820868 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.820902 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260620 14:24:19.821208 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.816796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40019 (local address 127.0.0.1:47990)
0620 14:24:19.816908 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.816948 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.817005 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.817440 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.817443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.817445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:19.817509 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.817511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.818831 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.818831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.820108 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.820111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.820975 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.820976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.821046 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.821049 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.821106 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.821148 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":55,"thread_start_us":61,"threads_started":1}
I20260620 14:24:19.821269 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.816910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.0.98.3:42381)
0620 14:24:19.817442 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.817448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.817458 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:19.817498 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.817500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.817517 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.817594 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:19.817786 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.818868 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.819975 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.820665 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.820740 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.820812 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.820897 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.820950 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":86,"threads_started":1}
I20260620 14:24:19.821485 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.817168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.0.98.2:40019)
0620 14:24:19.817292 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.817296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.817304 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:19.817334 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.817336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.817348 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.817547 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:19.817626 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.818753 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.820205 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.820894 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.820961 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.821031 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.821319 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.821365 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260620 14:24:19.821914 1921 tablet_service.cc:1467] Processing CreateTablet for tablet c2346355a5344c0b9c92ca926b58f07e (DEFAULT_TABLE table=test-table [id=880c34338f9b4c4baedfa0fca6ed64af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:19.822129 1781 tablet_service.cc:1467] Processing CreateTablet for tablet c2346355a5344c0b9c92ca926b58f07e (DEFAULT_TABLE table=test-table [id=880c34338f9b4c4baedfa0fca6ed64af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:19.822474 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.817393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.0.98.1:37375)
0620 14:24:19.817700 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.817704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.817714 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:19.817752 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.817754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.817771 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.817881 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:19.817968 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.818689 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.821069 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.821992 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.822071 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.822140 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.822291 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.822375 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":71,"threads_started":1}
I20260620 14:24:19.822530 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2346355a5344c0b9c92ca926b58f07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:19.822639 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2346355a5344c0b9c92ca926b58f07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:19.822669 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.817003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37375 (local address 127.0.0.1:34566)
0620 14:24:19.817121 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.817130 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.817174 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.817878 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.817879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.817881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:19.817912 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.817914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.818770 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.818771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.820108 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.820111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.822081 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.822082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.822197 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.822202 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.822270 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.822442 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":70,"thread_start_us":86,"threads_started":1}
I20260620 14:24:19.823374 1641 tablet_service.cc:1467] Processing CreateTablet for tablet c2346355a5344c0b9c92ca926b58f07e (DEFAULT_TABLE table=test-table [id=880c34338f9b4c4baedfa0fca6ed64af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:19.823710 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2346355a5344c0b9c92ca926b58f07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:19.826520 2010 tablet_bootstrap.cc:492] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d: Bootstrap starting.
I20260620 14:24:19.826663 2011 tablet_bootstrap.cc:492] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973: Bootstrap starting.
I20260620 14:24:19.827291 2010 tablet_bootstrap.cc:654] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:19.827291 2011 tablet_bootstrap.cc:654] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:19.827633 2010 log.cc:824] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:19.827641 2011 log.cc:824] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:19.827852 2012 tablet_bootstrap.cc:492] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822: Bootstrap starting.
I20260620 14:24:19.828353 2011 tablet_bootstrap.cc:492] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973: No bootstrap required, opened a new log
I20260620 14:24:19.828428 2011 ts_tablet_manager.cc:1397] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:19.828507 2012 tablet_bootstrap.cc:654] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:19.828609 2010 tablet_bootstrap.cc:492] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d: No bootstrap required, opened a new log
I20260620 14:24:19.828671 2010 ts_tablet_manager.cc:1397] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:19.828835 2012 log.cc:824] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:19.829478 2012 tablet_bootstrap.cc:492] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822: No bootstrap required, opened a new log
I20260620 14:24:19.829556 2012 ts_tablet_manager.cc:1397] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:19.830642 2010 raft_consensus.cc:348] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } }
I20260620 14:24:19.830657 2011 raft_consensus.cc:348] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } }
I20260620 14:24:19.830811 2011 raft_consensus.cc:374] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:19.830811 2010 raft_consensus.cc:374] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:19.830849 2011 raft_consensus.cc:729] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826d6a469821435bb5223b16b7d0b973, State: Initialized, Role: FOLLOWER
I20260620 14:24:19.830853 2010 raft_consensus.cc:729] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593713a3f89e466e978afd3f8afa5e7d, State: Initialized, Role: FOLLOWER
I20260620 14:24:19.830983 2010 consensus_queue.cc:260] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } }
I20260620 14:24:19.830983 2011 consensus_queue.cc:260] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } }
I20260620 14:24:19.831070 2012 raft_consensus.cc:348] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } }
I20260620 14:24:19.831220 2012 raft_consensus.cc:374] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:19.831247 2010 ts_tablet_manager.cc:1428] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:24:19.831247 2011 ts_tablet_manager.cc:1428] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:24:19.831259 2012 raft_consensus.cc:729] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0e6ac155b7470eb6e1e18f46ac2822, State: Initialized, Role: FOLLOWER
I20260620 14:24:19.831360 1847 heartbeater.cc:503] Master 127.0.98.62:46589 was elected leader, sending a full tablet report...
I20260620 14:24:19.831375 1987 heartbeater.cc:503] Master 127.0.98.62:46589 was elected leader, sending a full tablet report...
I20260620 14:24:19.831369 2012 consensus_queue.cc:260] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } }
I20260620 14:24:19.831619 2012 ts_tablet_manager.cc:1428] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:19.831765 1707 heartbeater.cc:503] Master 127.0.98.62:46589 was elected leader, sending a full tablet report...
W20260620 14:24:19.851243 1848 tablet.cc:2369] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:19.865083 2017 raft_consensus.cc:482] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:19.865183 2017 raft_consensus.cc:504] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } }
I20260620 14:24:19.865478 2017 leader_election.cc:288] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826d6a469821435bb5223b16b7d0b973 (127.0.98.2:40019), be0e6ac155b7470eb6e1e18f46ac2822 (127.0.98.1:37375)
I20260620 14:24:19.868407 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.865679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40019 (local address 127.0.98.3:49711)
0620 14:24:19.865743 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.865755 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.865798 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.865989 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.865990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.865992 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:19.866061 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.866063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.866848 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.866849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.867583 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.867585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.868142 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.868143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.868216 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.868219 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.868286 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.868326 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":2}
I20260620 14:24:19.868395 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.865765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49711 (local address 127.0.98.2:40019)
0620 14:24:19.865840 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.865843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.865852 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:24:19.865881 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.865882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.865893 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.865957 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:19.866133 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.866765 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.867678 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.868053 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.868132 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.868142 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.868292 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.868330 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:24:19.868652 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.865925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43983 (local address 127.0.98.1:37375)
0620 14:24:19.865976 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.865979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.866182 (+ 203us) server_negotiation.cc:407] Connection header received
0620 14:24:19.866250 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.866251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.866262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.866338 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:19.866514 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.866978 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.867891 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.868257 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.868340 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.868349 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.868532 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.868582 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:24:19.868881 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.865808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37375 (local address 127.0.98.3:43983)
0620 14:24:19.866009 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.866018 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.866183 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.866359 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.866360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.866361 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:19.866435 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.866436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.867065 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.867066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.867798 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.867800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.868342 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.868343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.868413 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.868417 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.868753 (+ 336us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.868795 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":126,"threads_started":1}
I20260620 14:24:19.868863 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2346355a5344c0b9c92ca926b58f07e" candidate_uuid: "593713a3f89e466e978afd3f8afa5e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826d6a469821435bb5223b16b7d0b973" is_pre_election: true
I20260620 14:24:19.869020 1801 raft_consensus.cc:2454] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593713a3f89e466e978afd3f8afa5e7d in term 0.
I20260620 14:24:19.869207 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2346355a5344c0b9c92ca926b58f07e" candidate_uuid: "593713a3f89e466e978afd3f8afa5e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" is_pre_election: true
I20260620 14:24:19.869357 1661 raft_consensus.cc:2454] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593713a3f89e466e978afd3f8afa5e7d in term 0.
I20260620 14:24:19.869349 1873 leader_election.cc:302] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593713a3f89e466e978afd3f8afa5e7d, 826d6a469821435bb5223b16b7d0b973; no voters:
I20260620 14:24:19.869489 2017 raft_consensus.cc:2781] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:19.869576 2017 raft_consensus.cc:482] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:19.869608 2017 raft_consensus.cc:3037] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:19.870591 2017 raft_consensus.cc:504] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } }
I20260620 14:24:19.870723 2017 leader_election.cc:288] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [CANDIDATE]: Term 1 election: Requested vote from peers 826d6a469821435bb5223b16b7d0b973 (127.0.98.2:40019), be0e6ac155b7470eb6e1e18f46ac2822 (127.0.98.1:37375)
I20260620 14:24:19.870935 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2346355a5344c0b9c92ca926b58f07e" candidate_uuid: "593713a3f89e466e978afd3f8afa5e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0e6ac155b7470eb6e1e18f46ac2822"
I20260620 14:24:19.870960 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2346355a5344c0b9c92ca926b58f07e" candidate_uuid: "593713a3f89e466e978afd3f8afa5e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826d6a469821435bb5223b16b7d0b973"
I20260620 14:24:19.871016 1661 raft_consensus.cc:3037] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:19.871033 1801 raft_consensus.cc:3037] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:19.871681 1801 raft_consensus.cc:2454] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593713a3f89e466e978afd3f8afa5e7d in term 1.
I20260620 14:24:19.871755 1661 raft_consensus.cc:2454] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593713a3f89e466e978afd3f8afa5e7d in term 1.
I20260620 14:24:19.871887 1873 leader_election.cc:302] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593713a3f89e466e978afd3f8afa5e7d, 826d6a469821435bb5223b16b7d0b973; no voters:
I20260620 14:24:19.872000 2017 raft_consensus.cc:2781] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:19.872144 2017 raft_consensus.cc:686] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 1 LEADER]: Becoming Leader. State: Replica: 593713a3f89e466e978afd3f8afa5e7d, State: Running, Role: LEADER
I20260620 14:24:19.872242 2017 consensus_queue.cc:237] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } }
I20260620 14:24:19.873143 1503 catalog_manager.cc:5495] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 593713a3f89e466e978afd3f8afa5e7d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "593713a3f89e466e978afd3f8afa5e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } health_report { overall_health: UNKNOWN } } }
I20260620 14:24:19.879212 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.876825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.0.98.3:42381)
0620 14:24:19.876869 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.876871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.876880 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:24:19.876966 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.876967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.876987 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.877054 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:19.877180 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.877617 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.878435 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.878618 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.878689 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.878895 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:19.878983 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:19.879060 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.879119 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.879136 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:24:19.880901 1661 raft_consensus.cc:1261] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822 [term 1 FOLLOWER]: Refusing update from remote peer 593713a3f89e466e978afd3f8afa5e7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:19.880998 1801 raft_consensus.cc:1261] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973 [term 1 FOLLOWER]: Refusing update from remote peer 593713a3f89e466e978afd3f8afa5e7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:19.881271 2017 consensus_queue.cc:1035] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:19.881412 2020 consensus_queue.cc:1035] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:19.883904 2025 mvcc.cc:205] Tried to move back new op lower bound from 7298930523669716992 to 7298930523637280768. Current Snapshot: MvccSnapshot[applied={T|T < 7298930523669716992}]
I20260620 14:24:19.884040 2023 mvcc.cc:205] Tried to move back new op lower bound from 7298930523669716992 to 7298930523637280768. Current Snapshot: MvccSnapshot[applied={T|T < 7298930523669716992}]
I20260620 14:24:19.884389 2024 mvcc.cc:205] Tried to move back new op lower bound from 7298930523669716992 to 7298930523637280768. Current Snapshot: MvccSnapshot[applied={T|T < 7298930523669716992}]
I20260620 14:24:19.887332 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:19.887413 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:19.888288 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a: Sending DeleteTablet for 3 replicas of tablet c2346355a5344c0b9c92ca926b58f07e
I20260620 14:24:19.888628 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet c2346355a5344c0b9c92ca926b58f07e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49674
I20260620 14:24:19.888710 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet c2346355a5344c0b9c92ca926b58f07e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47990
I20260620 14:24:19.888815 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet c2346355a5344c0b9c92ca926b58f07e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34566
I20260620 14:24:19.889252 2033 tablet_replica.cc:330] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973: stopping tablet replica
I20260620 14:24:19.889330 2034 tablet_replica.cc:330] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822: stopping tablet replica
I20260620 14:24:19.889313 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:24:19.889432 2034 raft_consensus.cc:2229] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 14:24:19.889467 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:19.889405 2033 raft_consensus.cc:2229] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:19.889607 2034 raft_consensus.cc:2258] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:19.889695 2033 raft_consensus.cc:2258] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:19.890121 2034 ts_tablet_manager.cc:1905] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:19.890230 2033 ts_tablet_manager.cc:1905] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:19.891220 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 360fc7d70ba04e039ca14969bad83650 (DEFAULT_TABLE table=test-table [id=47dd0417c76840deb7ae1b25b88b52fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:19.891294 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 360fc7d70ba04e039ca14969bad83650 (DEFAULT_TABLE table=test-table [id=47dd0417c76840deb7ae1b25b88b52fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:19.891341 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360fc7d70ba04e039ca14969bad83650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:19.891399 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360fc7d70ba04e039ca14969bad83650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:19.892387 2034 ts_tablet_manager.cc:1918] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:19.892454 2034 log.cc:1196] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/wal/wals/c2346355a5344c0b9c92ca926b58f07e
I20260620 14:24:19.892726 2034 ts_tablet_manager.cc:1939] T c2346355a5344c0b9c92ca926b58f07e P be0e6ac155b7470eb6e1e18f46ac2822: Deleting consensus metadata
I20260620 14:24:19.892877 2011 tablet_bootstrap.cc:492] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973: Bootstrap starting.
I20260620 14:24:19.893175 1489 catalog_manager.cc:4841] TS be0e6ac155b7470eb6e1e18f46ac2822 (127.0.98.1:37375): tablet c2346355a5344c0b9c92ca926b58f07e (table test-table [id=880c34338f9b4c4baedfa0fca6ed64af]) successfully deleted
I20260620 14:24:19.893265 2010 tablet_bootstrap.cc:492] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d: Bootstrap starting.
I20260620 14:24:19.893656 2011 tablet_bootstrap.cc:654] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:19.894088 2010 tablet_bootstrap.cc:654] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:19.894315 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 360fc7d70ba04e039ca14969bad83650 (DEFAULT_TABLE table=test-table [id=47dd0417c76840deb7ae1b25b88b52fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:19.894438 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360fc7d70ba04e039ca14969bad83650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:19.894675 2011 tablet_bootstrap.cc:492] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973: No bootstrap required, opened a new log
I20260620 14:24:19.894731 2011 ts_tablet_manager.cc:1397] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:24:19.894904 2011 raft_consensus.cc:348] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:19.894999 2011 raft_consensus.cc:374] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:19.895030 2011 raft_consensus.cc:729] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826d6a469821435bb5223b16b7d0b973, State: Initialized, Role: FOLLOWER
I20260620 14:24:19.895089 2033 ts_tablet_manager.cc:1918] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:19.895126 2011 consensus_queue.cc:260] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:19.895180 2033 log.cc:1196] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/c2346355a5344c0b9c92ca926b58f07e
I20260620 14:24:19.895274 2011 ts_tablet_manager.cc:1428] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:19.895527 2033 ts_tablet_manager.cc:1939] T c2346355a5344c0b9c92ca926b58f07e P 826d6a469821435bb5223b16b7d0b973: Deleting consensus metadata
I20260620 14:24:19.896032 1489 catalog_manager.cc:4841] TS 826d6a469821435bb5223b16b7d0b973 (127.0.98.2:40019): tablet c2346355a5344c0b9c92ca926b58f07e (table test-table [id=880c34338f9b4c4baedfa0fca6ed64af]) successfully deleted
I20260620 14:24:19.896147 2012 tablet_bootstrap.cc:492] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822: Bootstrap starting.
I20260620 14:24:19.896919 2012 tablet_bootstrap.cc:654] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:19.897069 2032 tablet_replica.cc:330] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d: stopping tablet replica
I20260620 14:24:19.897261 2032 raft_consensus.cc:2229] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:19.897437 2032 raft_consensus.cc:2258] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:19.897868 2012 tablet_bootstrap.cc:492] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822: No bootstrap required, opened a new log
I20260620 14:24:19.897909 2032 ts_tablet_manager.cc:1905] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:19.897977 2012 ts_tablet_manager.cc:1397] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:19.898597 2012 raft_consensus.cc:348] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:19.898794 2012 raft_consensus.cc:374] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:19.898895 2012 raft_consensus.cc:729] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0e6ac155b7470eb6e1e18f46ac2822, State: Initialized, Role: FOLLOWER
I20260620 14:24:19.899085 2012 consensus_queue.cc:260] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:19.899235 2012 ts_tablet_manager.cc:1428] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:19.900188 2032 ts_tablet_manager.cc:1918] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:19.900277 2032 log.cc:1196] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/c2346355a5344c0b9c92ca926b58f07e
I20260620 14:24:19.900502 2032 ts_tablet_manager.cc:1939] T c2346355a5344c0b9c92ca926b58f07e P 593713a3f89e466e978afd3f8afa5e7d: Deleting consensus metadata
I20260620 14:24:19.900882 1488 catalog_manager.cc:4841] TS 593713a3f89e466e978afd3f8afa5e7d (127.0.98.3:42381): tablet c2346355a5344c0b9c92ca926b58f07e (table test-table [id=880c34338f9b4c4baedfa0fca6ed64af]) successfully deleted
I20260620 14:24:19.902727 2010 tablet_bootstrap.cc:492] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d: No bootstrap required, opened a new log
I20260620 14:24:19.902767 2010 ts_tablet_manager.cc:1397] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260620 14:24:19.902938 2010 raft_consensus.cc:348] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:19.902995 2010 raft_consensus.cc:374] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:19.903009 2010 raft_consensus.cc:729] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593713a3f89e466e978afd3f8afa5e7d, State: Initialized, Role: FOLLOWER
I20260620 14:24:19.903062 2010 consensus_queue.cc:260] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:19.903163 2010 ts_tablet_manager.cc:1428] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260620 14:24:19.947131 1708 tablet.cc:2369] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:19.962513 2018 raft_consensus.cc:482] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:19.962622 2018 raft_consensus.cc:504] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:19.962972 2018 leader_election.cc:288] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826d6a469821435bb5223b16b7d0b973 (127.0.98.2:40019), 593713a3f89e466e978afd3f8afa5e7d (127.0.98.3:42381)
I20260620 14:24:19.966679 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.963142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40019 (local address 127.0.98.1:42889)
0620 14:24:19.963318 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.963332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.963392 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.963614 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.963615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.963617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:19.963743 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.963745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.964658 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.964658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.965604 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.965606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.966375 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.966377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.966474 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.966479 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.966544 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.966595 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":77,"threads_started":1}
I20260620 14:24:19.967043 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.963249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42889 (local address 127.0.98.2:40019)
0620 14:24:19.963360 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.963363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.963378 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:24:19.963502 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.963503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.963513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.963546 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:19.963829 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.964572 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.965717 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.966256 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.966371 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.966392 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.966960 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.966997 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:24:19.967439 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360fc7d70ba04e039ca14969bad83650" candidate_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826d6a469821435bb5223b16b7d0b973" is_pre_election: true
I20260620 14:24:19.967554 1801 raft_consensus.cc:2454] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be0e6ac155b7470eb6e1e18f46ac2822 in term 0.
I20260620 14:24:19.967535 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.963427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49143 (local address 127.0.98.3:42381)
0620 14:24:19.963476 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:19.963478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:19.963549 (+ 71us) server_negotiation.cc:407] Connection header received
0620 14:24:19.963583 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:19.963584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:19.963593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:19.963661 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:19.964684 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.965408 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.966572 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.967110 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.967184 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.967195 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:19.967433 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 14:24:19.967474 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:24:19.967844 2038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:19.963171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42381 (local address 127.0.98.1:49143)
0620 14:24:19.963470 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:19.963480 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:19.963537 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:19.963668 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:19.963669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:19.963670 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:19.963744 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.963745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.965498 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.965499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.966479 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:19.966481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:19.967214 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:19.967216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:19.967337 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:19.967342 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:24:19.967635 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 14:24:19.967700 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":58,"threads_started":1}
I20260620 14:24:19.967868 1593 leader_election.cc:302] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826d6a469821435bb5223b16b7d0b973, be0e6ac155b7470eb6e1e18f46ac2822; no voters:
I20260620 14:24:19.967996 2018 raft_consensus.cc:2781] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:19.968034 2018 raft_consensus.cc:482] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:19.968055 2018 raft_consensus.cc:3037] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:19.968282 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360fc7d70ba04e039ca14969bad83650" candidate_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593713a3f89e466e978afd3f8afa5e7d" is_pre_election: true
I20260620 14:24:19.968461 1941 raft_consensus.cc:2454] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be0e6ac155b7470eb6e1e18f46ac2822 in term 0.
I20260620 14:24:19.969041 2018 raft_consensus.cc:504] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:19.969205 2018 leader_election.cc:288] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [CANDIDATE]: Term 1 election: Requested vote from peers 826d6a469821435bb5223b16b7d0b973 (127.0.98.2:40019), 593713a3f89e466e978afd3f8afa5e7d (127.0.98.3:42381)
I20260620 14:24:19.969419 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360fc7d70ba04e039ca14969bad83650" candidate_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826d6a469821435bb5223b16b7d0b973"
I20260620 14:24:19.969425 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360fc7d70ba04e039ca14969bad83650" candidate_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593713a3f89e466e978afd3f8afa5e7d"
I20260620 14:24:19.969501 1801 raft_consensus.cc:3037] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:19.969519 1941 raft_consensus.cc:3037] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:19.970490 1941 raft_consensus.cc:2454] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be0e6ac155b7470eb6e1e18f46ac2822 in term 1.
I20260620 14:24:19.970497 1801 raft_consensus.cc:2454] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be0e6ac155b7470eb6e1e18f46ac2822 in term 1.
I20260620 14:24:19.970665 1593 leader_election.cc:302] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826d6a469821435bb5223b16b7d0b973, be0e6ac155b7470eb6e1e18f46ac2822; no voters:
I20260620 14:24:19.970788 2018 raft_consensus.cc:2781] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:19.970920 2018 raft_consensus.cc:686] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 1 LEADER]: Becoming Leader. State: Replica: be0e6ac155b7470eb6e1e18f46ac2822, State: Running, Role: LEADER
I20260620 14:24:19.970994 2018 consensus_queue.cc:237] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:19.971761 1504 catalog_manager.cc:5495] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 reported cstate change: term changed from 0 to 1, leader changed from <none> to be0e6ac155b7470eb6e1e18f46ac2822 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:24:20 dist-test-slave-hfc3 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:20.015934 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:20.007223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46589 (local address 127.0.0.1:41482)
0620 14:24:20.007432 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:20.007446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:20.007624 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:20.007790 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:20.007791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:20.008203 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:20.008261 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:20.008263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.009139 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.009140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:20.009974 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:20.009976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.010558 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.010559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:20.010621 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:20.011381 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:20.011394 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:20.013164 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:20.014886 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:20.014894 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:20.014896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:20.015084 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:20.015313 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:20.015313 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:20.015314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:20.015370 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:20.015616 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:20.015622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:20.015667 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:24:20.015760 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:24:20.015861 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":122,"threads_started":1}
I20260620 14:24:20.016050 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:20.007287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.0.98.62:46589)
0620 14:24:20.007373 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:20.007375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:20.007638 (+ 263us) server_negotiation.cc:407] Connection header received
0620 14:24:20.007702 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:20.007703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:20.007712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:20.007761 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:20.008349 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.009048 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.010083 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.010289 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.010536 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:20.013275 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:20.013277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:20.013277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:20.013284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:20.014789 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:20.015190 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:20.015191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:20.015192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:20.015248 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:20.015441 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:20.015442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:20.015442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:20.015551 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:20.015625 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:20.015841 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 14:24:20.015926 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:24:20.018460 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41482:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:24:20.018642 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260620 14:24:20.018884 1988 tablet.cc:2369] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:20.020212 1641 tablet_service.cc:1467] Processing CreateTablet for tablet a87c77a90b1b4e64bcde0156927e9a1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f331f63273a4aada492eec1bd0bd886]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:24:20.020289 1781 tablet_service.cc:1467] Processing CreateTablet for tablet a87c77a90b1b4e64bcde0156927e9a1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f331f63273a4aada492eec1bd0bd886]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:24:20.020380 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87c77a90b1b4e64bcde0156927e9a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:20.020408 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87c77a90b1b4e64bcde0156927e9a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:20.020358 1921 tablet_service.cc:1467] Processing CreateTablet for tablet a87c77a90b1b4e64bcde0156927e9a1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f331f63273a4aada492eec1bd0bd886]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:24:20.020638 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87c77a90b1b4e64bcde0156927e9a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:20.022115 2011 tablet_bootstrap.cc:492] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973: Bootstrap starting.
I20260620 14:24:20.022120 2012 tablet_bootstrap.cc:492] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: Bootstrap starting.
I20260620 14:24:20.022547 2010 tablet_bootstrap.cc:492] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d: Bootstrap starting.
I20260620 14:24:20.022979 2010 tablet_bootstrap.cc:654] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:20.023059 2011 tablet_bootstrap.cc:654] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:20.023059 2012 tablet_bootstrap.cc:654] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:20.024057 2010 tablet_bootstrap.cc:492] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d: No bootstrap required, opened a new log
I20260620 14:24:20.024103 2010 ts_tablet_manager.cc:1397] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:20.024161 2011 tablet_bootstrap.cc:492] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973: No bootstrap required, opened a new log
I20260620 14:24:20.024219 2011 ts_tablet_manager.cc:1397] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:24:20.024225 2010 raft_consensus.cc:348] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:20.024288 2010 raft_consensus.cc:374] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:20.024307 2010 raft_consensus.cc:729] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593713a3f89e466e978afd3f8afa5e7d, State: Initialized, Role: FOLLOWER
I20260620 14:24:20.024345 2010 consensus_queue.cc:260] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:20.024423 2010 ts_tablet_manager.cc:1428] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:20.024489 2012 tablet_bootstrap.cc:492] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: No bootstrap required, opened a new log
I20260620 14:24:20.024554 2012 ts_tablet_manager.cc:1397] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:24:20.024544 2011 raft_consensus.cc:348] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:20.024643 2011 raft_consensus.cc:374] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:20.024685 2011 raft_consensus.cc:729] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826d6a469821435bb5223b16b7d0b973, State: Initialized, Role: FOLLOWER
I20260620 14:24:20.024770 2011 consensus_queue.cc:260] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:20.024854 2011 ts_tablet_manager.cc:1428] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:20.024813 2012 raft_consensus.cc:348] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:20.024930 2012 raft_consensus.cc:374] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:20.024956 2012 raft_consensus.cc:729] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0e6ac155b7470eb6e1e18f46ac2822, State: Initialized, Role: FOLLOWER
I20260620 14:24:20.024993 2012 consensus_queue.cc:260] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:20.025097 2012 ts_tablet_manager.cc:1428] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:20.025184 2020 tablet_replica.cc:439] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } }
I20260620 14:24:20.025167 2018 tablet_replica.cc:439] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } }
I20260620 14:24:20.025250 2020 tablet_replica.cc:442] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:20.025252 2018 tablet_replica.cc:442] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:20.025403 2016 tablet_replica.cc:439] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } }
I20260620 14:24:20.025533 2016 tablet_replica.cc:442] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:20.155305 2018 raft_consensus.cc:482] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:20.155417 2018 raft_consensus.cc:504] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:20.155637 2018 leader_election.cc:288] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826d6a469821435bb5223b16b7d0b973 (127.0.98.2:40019), 593713a3f89e466e978afd3f8afa5e7d (127.0.98.3:42381)
I20260620 14:24:20.155951 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87c77a90b1b4e64bcde0156927e9a1d" candidate_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593713a3f89e466e978afd3f8afa5e7d" is_pre_election: true
I20260620 14:24:20.155951 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87c77a90b1b4e64bcde0156927e9a1d" candidate_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826d6a469821435bb5223b16b7d0b973" is_pre_election: true
I20260620 14:24:20.156075 1801 raft_consensus.cc:2454] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be0e6ac155b7470eb6e1e18f46ac2822 in term 0.
I20260620 14:24:20.156075 1941 raft_consensus.cc:2454] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be0e6ac155b7470eb6e1e18f46ac2822 in term 0.
I20260620 14:24:20.156283 1592 leader_election.cc:302] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593713a3f89e466e978afd3f8afa5e7d, be0e6ac155b7470eb6e1e18f46ac2822; no voters:
I20260620 14:24:20.156437 2018 raft_consensus.cc:2781] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:20.156500 2018 raft_consensus.cc:482] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:20.156533 2018 raft_consensus.cc:3037] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:20.157428 2018 raft_consensus.cc:504] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:20.157577 2018 leader_election.cc:288] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [CANDIDATE]: Term 1 election: Requested vote from peers 826d6a469821435bb5223b16b7d0b973 (127.0.98.2:40019), 593713a3f89e466e978afd3f8afa5e7d (127.0.98.3:42381)
I20260620 14:24:20.157733 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87c77a90b1b4e64bcde0156927e9a1d" candidate_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593713a3f89e466e978afd3f8afa5e7d"
I20260620 14:24:20.157828 1941 raft_consensus.cc:3037] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:20.157809 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87c77a90b1b4e64bcde0156927e9a1d" candidate_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826d6a469821435bb5223b16b7d0b973"
I20260620 14:24:20.157927 1801 raft_consensus.cc:3037] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:20.158560 1941 raft_consensus.cc:2454] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be0e6ac155b7470eb6e1e18f46ac2822 in term 1.
I20260620 14:24:20.158758 1801 raft_consensus.cc:2454] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be0e6ac155b7470eb6e1e18f46ac2822 in term 1.
I20260620 14:24:20.158783 1592 leader_election.cc:302] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593713a3f89e466e978afd3f8afa5e7d, be0e6ac155b7470eb6e1e18f46ac2822; no voters:
I20260620 14:24:20.158874 2018 raft_consensus.cc:2781] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:20.158921 2018 raft_consensus.cc:686] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [term 1 LEADER]: Becoming Leader. State: Replica: be0e6ac155b7470eb6e1e18f46ac2822, State: Running, Role: LEADER
I20260620 14:24:20.158979 2018 consensus_queue.cc:237] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } }
I20260620 14:24:20.159294 2040 tablet_replica.cc:439] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: TxnStatusTablet state changed. Reason: New leader be0e6ac155b7470eb6e1e18f46ac2822. Latest consensus state: current_term: 1 leader_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } }
I20260620 14:24:20.159471 2040 tablet_replica.cc:442] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:24:20.159724 2052 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:24:20.159852 2052 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:24:20.159739 1504 catalog_manager.cc:5495] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 reported cstate change: term changed from 0 to 1, leader changed from <none> to be0e6ac155b7470eb6e1e18f46ac2822 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } health_report { overall_health: UNKNOWN } } }
I20260620 14:24:20.172863 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:20.169107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.0.98.1:37375)
0620 14:24:20.169148 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:20.169151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:20.169302 (+ 151us) server_negotiation.cc:407] Connection header received
0620 14:24:20.169395 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:20.169396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:20.169412 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:20.169473 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:20.169674 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.170190 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.171603 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.172125 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.172216 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:20.172438 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:20.172551 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:20.172634 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:20.172687 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:24:20.172710 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:24:20.172830 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:20.169078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37375 (local address 127.0.0.1:34568)
0620 14:24:20.169195 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:20.169218 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:20.169301 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:20.169507 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:20.169508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:20.169510 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:24:20.169577 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:20.169580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.170290 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.170291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:20.171460 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:20.171465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.172260 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.172261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:20.172324 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:20.172338 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:24:20.172630 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:24:20.172639 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:24:20.172684 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:24:20.172732 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 14:24:20.174683 1801 raft_consensus.cc:1261] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973 [term 1 FOLLOWER]: Refusing update from remote peer be0e6ac155b7470eb6e1e18f46ac2822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:20.174810 1941 raft_consensus.cc:1261] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d [term 1 FOLLOWER]: Refusing update from remote peer be0e6ac155b7470eb6e1e18f46ac2822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:20.175031 2040 consensus_queue.cc:1035] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:20.175185 2018 consensus_queue.cc:1035] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:20.175933 2020 tablet_replica.cc:439] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d: TxnStatusTablet state changed. Reason: New leader be0e6ac155b7470eb6e1e18f46ac2822. Latest consensus state: current_term: 1 leader_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } }
I20260620 14:24:20.175938 2016 tablet_replica.cc:439] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973: TxnStatusTablet state changed. Reason: New leader be0e6ac155b7470eb6e1e18f46ac2822. Latest consensus state: current_term: 1 leader_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } }
I20260620 14:24:20.176033 2020 tablet_replica.cc:442] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:20.176033 2016 tablet_replica.cc:442] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:20.176995 2018 tablet_replica.cc:439] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } }
I20260620 14:24:20.177085 2018 tablet_replica.cc:442] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:24:20.177042 2040 tablet_replica.cc:439] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } }
I20260620 14:24:20.177117 2040 tablet_replica.cc:442] T a87c77a90b1b4e64bcde0156927e9a1d P be0e6ac155b7470eb6e1e18f46ac2822: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:24:20.177386 2020 tablet_replica.cc:439] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } }
I20260620 14:24:20.177475 2020 tablet_replica.cc:442] T a87c77a90b1b4e64bcde0156927e9a1d P 593713a3f89e466e978afd3f8afa5e7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:20.177435 2016 tablet_replica.cc:439] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e6ac155b7470eb6e1e18f46ac2822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37375 } } peers { permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 } } peers { permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } }
I20260620 14:24:20.177518 2016 tablet_replica.cc:442] T a87c77a90b1b4e64bcde0156927e9a1d P 826d6a469821435bb5223b16b7d0b973: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:20.181998 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:20.179201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.0.98.1:37375)
0620 14:24:20.179250 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:20.179252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:20.179262 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:20.179324 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:20.179325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:20.179336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:20.179402 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:20.179541 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.180406 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.181174 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.181517 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.181593 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:20.181721 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:20.181757 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:20.181839 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:20.181899 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:24:20.181919 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:24:20.188723 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:20.185151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49139 (local address 127.0.98.1:37375)
0620 14:24:20.185465 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:20.185467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:20.185610 (+ 143us) server_negotiation.cc:407] Connection header received
0620 14:24:20.185635 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:20.185636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:20.185646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:20.185732 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:20.185939 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.186800 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.187752 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.188082 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.188160 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:20.188386 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:20.188439 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:20.188503 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:20.188615 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:24:20.188634 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260620 14:24:20.188845 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:20.185123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37375 (local address 127.0.98.1:49139)
0620 14:24:20.185480 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:20.185490 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:20.185598 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:20.185734 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:20.185734 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:20.185736 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:24:20.185867 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:20.185868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.186878 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.186879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:20.187663 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:20.187665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.188166 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.188167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:20.188262 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:20.188301 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:24:20.188532 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:24:20.188535 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:20.188595 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:24:20.188639 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":248,"threads_started":1}
I20260620 14:24:20.191346 1801 raft_consensus.cc:1261] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973 [term 1 FOLLOWER]: Refusing update from remote peer be0e6ac155b7470eb6e1e18f46ac2822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:20.191346 1941 raft_consensus.cc:1261] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d [term 1 FOLLOWER]: Refusing update from remote peer be0e6ac155b7470eb6e1e18f46ac2822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:20.191563 2018 consensus_queue.cc:1035] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826d6a469821435bb5223b16b7d0b973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:20.191771 2040 consensus_queue.cc:1035] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593713a3f89e466e978afd3f8afa5e7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:20.204350 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:20.204473 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:20.205080 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d9de597d2064ca5a50b470f19484a6a: Sending DeleteTablet for 3 replicas of tablet 360fc7d70ba04e039ca14969bad83650
I20260620 14:24:20.205343 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 360fc7d70ba04e039ca14969bad83650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47990
I20260620 14:24:20.205386 1640 tablet_service.cc:1514] Processing DeleteTablet for tablet 360fc7d70ba04e039ca14969bad83650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34566
I20260620 14:24:20.205468 2033 tablet_replica.cc:330] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973: stopping tablet replica
I20260620 14:24:20.205520 2033 raft_consensus.cc:2229] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:20.205524 2034 tablet_replica.cc:330] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822: stopping tablet replica
I20260620 14:24:20.205571 2033 raft_consensus.cc:2258] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:20.205587 2034 raft_consensus.cc:2229] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:20.205685 2034 raft_consensus.cc:2258] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:20.205909 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 360fc7d70ba04e039ca14969bad83650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49674
I20260620 14:24:20.206037 2032 tablet_replica.cc:330] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d: stopping tablet replica
I20260620 14:24:20.206092 2032 raft_consensus.cc:2229] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:20.206130 2032 raft_consensus.cc:2258] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:20.206375 2033 ts_tablet_manager.cc:1905] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:20.206382 2034 ts_tablet_manager.cc:1905] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:20.206488 2032 ts_tablet_manager.cc:1905] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:20.208099 2034 ts_tablet_manager.cc:1918] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:20.208109 2032 ts_tablet_manager.cc:1918] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:20.208109 2033 ts_tablet_manager.cc:1918] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:20.208200 2032 log.cc:1196] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/360fc7d70ba04e039ca14969bad83650
I20260620 14:24:20.208204 2034 log.cc:1196] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-0/wal/wals/360fc7d70ba04e039ca14969bad83650
I20260620 14:24:20.208200 2033 log.cc:1196] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/360fc7d70ba04e039ca14969bad83650
I20260620 14:24:20.208472 2034 ts_tablet_manager.cc:1939] T 360fc7d70ba04e039ca14969bad83650 P be0e6ac155b7470eb6e1e18f46ac2822: Deleting consensus metadata
I20260620 14:24:20.208470 2033 ts_tablet_manager.cc:1939] T 360fc7d70ba04e039ca14969bad83650 P 826d6a469821435bb5223b16b7d0b973: Deleting consensus metadata
I20260620 14:24:20.208470 2032 ts_tablet_manager.cc:1939] T 360fc7d70ba04e039ca14969bad83650 P 593713a3f89e466e978afd3f8afa5e7d: Deleting consensus metadata
I20260620 14:24:20.208897 1489 catalog_manager.cc:4841] TS be0e6ac155b7470eb6e1e18f46ac2822 (127.0.98.1:37375): tablet 360fc7d70ba04e039ca14969bad83650 (table test-table [id=47dd0417c76840deb7ae1b25b88b52fb]) successfully deleted
I20260620 14:24:20.208917 1488 catalog_manager.cc:4841] TS 593713a3f89e466e978afd3f8afa5e7d (127.0.98.3:42381): tablet 360fc7d70ba04e039ca14969bad83650 (table test-table [id=47dd0417c76840deb7ae1b25b88b52fb]) successfully deleted
I20260620 14:24:20.209000 1489 catalog_manager.cc:4841] TS 826d6a469821435bb5223b16b7d0b973 (127.0.98.2:40019): tablet 360fc7d70ba04e039ca14969bad83650 (table test-table [id=47dd0417c76840deb7ae1b25b88b52fb]) successfully deleted
Jun 20 14:24:20 dist-test-slave-hfc3 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965459, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:24:20.219053 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:20.208876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.0.98.1:37375)
0620 14:24:20.208916 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:20.208919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:20.209191 (+ 272us) server_negotiation.cc:407] Connection header received
0620 14:24:20.209261 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:20.209262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:20.209273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:20.209347 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:20.209984 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.210863 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.211817 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.212152 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.212238 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:20.215259 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:20.215268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:20.215269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:20.215277 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:20.217503 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:20.217876 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:20.217880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:20.217881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:20.217951 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:20.218141 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:20.218142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:20.218142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:20.218374 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:20.218742 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:20.218788 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 14:24:20.218878 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:24:20.219375 1699 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34588
I20260620 14:24:20.226570 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:20.220459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.0.98.62:46589)
0620 14:24:20.220534 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:20.220537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:20.220546 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:24:20.220571 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:20.220572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:20.220582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:20.220654 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:20.221213 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.221707 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.222484 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:20.222679 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:20.222763 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:20.223958 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:20.223959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:20.223959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:20.223966 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:20.225408 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:20.225764 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:20.225765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:20.225766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:20.225823 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:20.226029 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:20.226030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:20.226031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:20.226138 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:20.226244 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:20.226385 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 14:24:20.226466 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 14:24:20.226811 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41490
I20260620 14:24:20.227442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 1576
I20260620 14:24:20.233728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 1716
I20260620 14:24:20.239750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 1856
I20260620 14:24:20.245816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 1472
2026-06-20T14:24:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2255 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:20 dist-test-slave-hfc3 krb5kdc[2070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:20 dist-test-slave-hfc3 krb5kdc[2070](info): set up 2 sockets
Jun 20 14:24:20 dist-test-slave-hfc3 krb5kdc[2070](info): commencing operation
krb5kdc: starting...
W20260620 14:24:21.290215 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:21 dist-test-slave-hfc3 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:21.408447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42431
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:21.489528 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:21.489723 2083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:21.489745 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:21.490935 2083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:21.490978 2083 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:24:21.490991 2083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:21.491005 2083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:21.491017 2083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:21.491065 2083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:21.492498 2083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42431
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:21.492738 2083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:21.492974 2083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:21.497793 2088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:21.497794 2091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:21.497785 2089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:21.498039 2083 server_base.cc:1034] running on GCE node
I20260620 14:24:21.498291 2083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:21.498672 2083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:21.499847 2083 hybrid_clock.cc:648] HybridClock initialized: now 1781965461499833 us; error 32 us; skew 500 ppm
Jun 20 14:24:21 dist-test-slave-hfc3 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:21.502681 2083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:21.503196 2083 webserver.cc:460] Webserver started at http://127.0.98.62:40639/ using document root <none> and password file <none>
I20260620 14:24:21.503455 2083 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:21.503527 2083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:21.503679 2083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:21.504769 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "cf3b0d2e20924e8eb2b2a749fd09a735"
format_stamp: "Formatted at 2026-06-20 14:24:21 on dist-test-slave-hfc3"
server_key: "4de3ff612c92f141b24874143db2eebc"
server_key_iv: "7161fee2c2772c21479545c18534fc40"
server_key_version: "encryptionkey@0"
I20260620 14:24:21.505154 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf3b0d2e20924e8eb2b2a749fd09a735"
format_stamp: "Formatted at 2026-06-20 14:24:21 on dist-test-slave-hfc3"
server_key: "4de3ff612c92f141b24874143db2eebc"
server_key_iv: "7161fee2c2772c21479545c18534fc40"
server_key_version: "encryptionkey@0"
I20260620 14:24:21.506640 2083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:24:21.507465 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:21.507643 2083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:21.507741 2083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "cf3b0d2e20924e8eb2b2a749fd09a735"
format_stamp: "Formatted at 2026-06-20 14:24:21 on dist-test-slave-hfc3"
server_key: "4de3ff612c92f141b24874143db2eebc"
server_key_iv: "7161fee2c2772c21479545c18534fc40"
server_key_version: "encryptionkey@0"
I20260620 14:24:21.507834 2083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:21.545183 2083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:21.547719 2083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:21.547909 2083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:21.552955 2083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42431
I20260620 14:24:21.552994 2160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42431 every 8 connection(s)
I20260620 14:24:21.553372 2083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:21.553879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 2083
I20260620 14:24:21.553990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:21.554155 2161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:21.554183 392 external_mini_cluster.cc:1428] Setting key 67c9d54b06b8db6b98625e3e1798c496
I20260620 14:24:21.557060 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735: Bootstrap starting.
I20260620 14:24:21.557725 2161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:21.558020 2161 log.cc:824] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:21.558753 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735: No bootstrap required, opened a new log
Jun 20 14:24:21 dist-test-slave-hfc3 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:21.561169 2161 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3b0d2e20924e8eb2b2a749fd09a735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42431 } }
I20260620 14:24:21.561314 2161 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:21.561336 2161 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf3b0d2e20924e8eb2b2a749fd09a735, State: Initialized, Role: FOLLOWER
I20260620 14:24:21.561503 2161 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3b0d2e20924e8eb2b2a749fd09a735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42431 } }
I20260620 14:24:21.561590 2161 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:21.561635 2161 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:21.561709 2161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:21.562518 2161 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3b0d2e20924e8eb2b2a749fd09a735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42431 } }
I20260620 14:24:21.562642 2161 leader_election.cc:302] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf3b0d2e20924e8eb2b2a749fd09a735; no voters:
I20260620 14:24:21.562819 2161 leader_election.cc:288] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:21.562873 2166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:21.563027 2166 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [term 1 LEADER]: Becoming Leader. State: Replica: cf3b0d2e20924e8eb2b2a749fd09a735, State: Running, Role: LEADER
I20260620 14:24:21.563092 2161 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:21.563123 2166 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3b0d2e20924e8eb2b2a749fd09a735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42431 } }
I20260620 14:24:21.563458 2166 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf3b0d2e20924e8eb2b2a749fd09a735. Latest consensus state: current_term: 1 leader_uuid: "cf3b0d2e20924e8eb2b2a749fd09a735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3b0d2e20924e8eb2b2a749fd09a735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42431 } } }
I20260620 14:24:21.563464 2167 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf3b0d2e20924e8eb2b2a749fd09a735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3b0d2e20924e8eb2b2a749fd09a735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42431 } } }
I20260620 14:24:21.563547 2166 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:21.563598 2167 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:21.563771 2173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:21.564334 2173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:21.565675 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:21.554787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.0.98.62:42431)
0620 14:24:21.554973 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:21.554982 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:21.555008 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:24:21.555644 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:21.555651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:21.555703 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:21.555861 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:21.556503 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.557450 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.558323 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.558768 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.558901 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:21.561625 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:21.561630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:21.561630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:21.561644 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:21.563855 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:21.564289 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:21.564293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:21.564293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:21.564395 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:21.564604 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:21.564605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:21.564605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:21.564980 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:21.565146 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:21.565296 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:24:21.565474 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260620 14:24:21.566282 2173 catalog_manager.cc:1347] Generated new cluster ID: cddbef5407304e61b499f2ce1850e795
I20260620 14:24:21.566349 2173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:21.588472 2173 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:21.589069 2173 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:21.594108 2173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735: Generated new TSK 0
I20260620 14:24:21.594380 2173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:21.632898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42431
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:21.715713 2187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:21.715917 2187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:21.715942 2187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:21.717012 2187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:21.717047 2187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:21.717067 2187 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:24:21.717079 2187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:24:21.717095 2187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:21.718472 2187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:21.718653 2187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:21.718817 2187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:21.723793 2195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:21.723850 2193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:21.723789 2192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:21.724080 2187 server_base.cc:1034] running on GCE node
I20260620 14:24:21.724269 2187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:21.724454 2187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:21.725596 2187 hybrid_clock.cc:648] HybridClock initialized: now 1781965461725585 us; error 28 us; skew 500 ppm
Jun 20 14:24:21 dist-test-slave-hfc3 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:21.728281 2187 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:21.728763 2187 webserver.cc:460] Webserver started at http://127.0.98.1:34387/ using document root <none> and password file <none>
I20260620 14:24:21.728998 2187 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:21.729046 2187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:21.729175 2187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:21.730213 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "3f8f1efc97f94c7a8777472e8f88a0de"
format_stamp: "Formatted at 2026-06-20 14:24:21 on dist-test-slave-hfc3"
server_key: "c58949adc0d235765b233a21fe613d09"
server_key_iv: "9992e19808ac7eca0fd987c04c862d84"
server_key_version: "encryptionkey@0"
I20260620 14:24:21.730633 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3f8f1efc97f94c7a8777472e8f88a0de"
format_stamp: "Formatted at 2026-06-20 14:24:21 on dist-test-slave-hfc3"
server_key: "c58949adc0d235765b233a21fe613d09"
server_key_iv: "9992e19808ac7eca0fd987c04c862d84"
server_key_version: "encryptionkey@0"
I20260620 14:24:21.731976 2187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:21.732698 2202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:21.732883 2187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:21.732982 2187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "3f8f1efc97f94c7a8777472e8f88a0de"
format_stamp: "Formatted at 2026-06-20 14:24:21 on dist-test-slave-hfc3"
server_key: "c58949adc0d235765b233a21fe613d09"
server_key_iv: "9992e19808ac7eca0fd987c04c862d84"
server_key_version: "encryptionkey@0"
I20260620 14:24:21.733063 2187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:21.753336 2187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:21.755374 2187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:21.755548 2187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:21.756114 2187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:21.756174 2187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:21.756217 2187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:21.756240 2187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:24:21 dist-test-slave-hfc3 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:21.763459 2187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34503
I20260620 14:24:21.763878 2187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:21.763850 2317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34503 every 8 connection(s)
I20260620 14:24:21.766196 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:21.756554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37017 (local address 127.0.98.62:42431)
0620 14:24:21.756631 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:21.756633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:21.756855 (+ 222us) server_negotiation.cc:407] Connection header received
0620 14:24:21.757341 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:21.757343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:21.757357 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:21.757445 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:21.758163 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.759071 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.760128 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.760362 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.760443 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:21.763204 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:21.763206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:21.763206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:21.763215 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:21.764757 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:21.765179 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:21.765181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:21.765182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:21.765243 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:21.765502 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:21.765503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:21.765504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:21.765630 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:21.765708 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:21.766015 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 14:24:21.766102 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:24:21.766410 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:21.756531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42431 (local address 127.0.98.1:37017)
0620 14:24:21.756714 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:21.756754 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:21.757241 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:21.757527 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:21.757529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:21.757826 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:21.758065 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:21.758071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.759190 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.759191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:21.760031 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:21.760033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.760452 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.760453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:21.760588 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:21.761113 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:21.761123 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:21.763107 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:21.764859 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:21.764866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:21.764868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:21.765096 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:21.765320 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:21.765321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:21.765321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:21.765426 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:21.765699 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:21.765705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:21.765861 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 14:24:21.765990 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:24:21.766151 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":118,"threads_started":1}
I20260620 14:24:21.766934 2318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42431
I20260620 14:24:21.767025 2318 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:21.767217 2318 heartbeater.cc:511] Master 127.0.98.62:42431 requested a full tablet report, sending...
I20260620 14:24:21.767777 2114 ts_manager.cc:194] Registered new tserver with Master: 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503)
I20260620 14:24:21.768205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 2187
I20260620 14:24:21.768322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:21.768537 392 external_mini_cluster.cc:1428] Setting key efa36387eaf81f5c7109100bd44b1723
I20260620 14:24:21.768770 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37017
I20260620 14:24:21.774147 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:21.767989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56157 (local address 127.0.98.62:42431)
0620 14:24:21.768032 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:21.768034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:21.768244 (+ 210us) server_negotiation.cc:407] Connection header received
0620 14:24:21.768294 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:21.768295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:21.768308 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:21.768354 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:21.768718 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.769491 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.770232 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.770508 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.770591 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:21.771517 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:21.771518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:21.771519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:21.771526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:21.772885 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:21.773276 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:21.773277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:21.773278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:21.773406 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:21.773630 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:21.773631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:21.773632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:21.773745 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:21.773813 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:21.773978 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:24:21.774055 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:24:21.774161 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:21.767945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42431 (local address 127.0.98.1:56157)
0620 14:24:21.768158 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:21.768169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:21.768237 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:21.768370 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:21.768370 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:21.768570 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:21.768630 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:21.768631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.769588 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.769589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:21.770135 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:21.770136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.770592 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.770593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:21.770676 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:21.771117 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:21.771121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:21.771390 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:21.773004 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:21.773005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:21.773005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:21.773186 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:21.773481 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:21.773482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:21.773482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:21.773550 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:21.773817 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:21.773819 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:21.773864 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:24:21.773951 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:24:21.774057 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":138,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:21.793962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42431
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:21.876178 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:21.876399 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:21.876426 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:21.877526 2327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:21.877562 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:21.877583 2327 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:24:21.877594 2327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:24:21.877611 2327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:21.879088 2327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:21.879274 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:21.879455 2327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:21.884516 2327 server_base.cc:1034] running on GCE node
W20260620 14:24:21.884482 2335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:21.884460 2333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:21.884454 2332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:21.884899 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:21.885185 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:21.886390 2327 hybrid_clock.cc:648] HybridClock initialized: now 1781965461886327 us; error 70 us; skew 500 ppm
Jun 20 14:24:21 dist-test-slave-hfc3 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:21.889109 2327 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:21.889571 2327 webserver.cc:460] Webserver started at http://127.0.98.2:32851/ using document root <none> and password file <none>
I20260620 14:24:21.889810 2327 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:21.889882 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:21.890043 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:21.891217 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "ec32d73bdb9d430c90b822b147fd1105"
format_stamp: "Formatted at 2026-06-20 14:24:21 on dist-test-slave-hfc3"
server_key: "79a291e94c10ac078c092dc52054b489"
server_key_iv: "572e81721d9d5866cee25cccc6ef50ed"
server_key_version: "encryptionkey@0"
I20260620 14:24:21.891599 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ec32d73bdb9d430c90b822b147fd1105"
format_stamp: "Formatted at 2026-06-20 14:24:21 on dist-test-slave-hfc3"
server_key: "79a291e94c10ac078c092dc52054b489"
server_key_iv: "572e81721d9d5866cee25cccc6ef50ed"
server_key_version: "encryptionkey@0"
I20260620 14:24:21.892978 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:24:21.893668 2342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:21.893867 2327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:21.893967 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "ec32d73bdb9d430c90b822b147fd1105"
format_stamp: "Formatted at 2026-06-20 14:24:21 on dist-test-slave-hfc3"
server_key: "79a291e94c10ac078c092dc52054b489"
server_key_iv: "572e81721d9d5866cee25cccc6ef50ed"
server_key_version: "encryptionkey@0"
I20260620 14:24:21.894045 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:21.909401 2327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:21.912055 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:21.912259 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:21.912873 2327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:21.912942 2327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:21.912981 2327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:21.913007 2327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:21.919778 2327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46237
I20260620 14:24:21.919857 2457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46237 every 8 connection(s)
I20260620 14:24:21.920204 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:24:21 dist-test-slave-hfc3 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:21.923306 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:21.913347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44951 (local address 127.0.98.62:42431)
0620 14:24:21.913469 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:21.913471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:21.913687 (+ 216us) server_negotiation.cc:407] Connection header received
0620 14:24:21.914150 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:21.914152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:21.914165 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:21.914247 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:21.915221 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.916096 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.917681 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.918060 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.918145 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:21.920847 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:21.920848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:21.920849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:21.920855 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:21.921981 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:21.922441 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:21.922442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:21.922443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:21.922511 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:21.922748 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:21.922748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:21.922749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:21.922853 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:21.922911 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:21.923176 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 14:24:21.923243 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 14:24:21.923463 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:21.913340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42431 (local address 127.0.98.2:44951)
0620 14:24:21.913545 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:21.913585 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:21.914059 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:21.914336 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:21.914341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:21.914800 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:21.915091 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:21.915097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.916218 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.916219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:21.917577 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:21.917581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.918165 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.918166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:21.918270 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:21.918967 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:21.918977 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:21.920753 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:21.922053 (+ 1300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:21.922059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:21.922060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:21.922281 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:21.922580 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:21.922580 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:21.922581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:21.922681 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:21.922919 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:21.922928 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:21.923058 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 14:24:21.923136 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:24:21.923254 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":109,"threads_started":1}
I20260620 14:24:21.923934 2458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42431
I20260620 14:24:21.924026 2458 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:21.924207 2458 heartbeater.cc:511] Master 127.0.98.62:42431 requested a full tablet report, sending...
I20260620 14:24:21.924556 2114 ts_manager.cc:194] Registered new tserver with Master: ec32d73bdb9d430c90b822b147fd1105 (127.0.98.2:46237)
I20260620 14:24:21.925076 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44951
I20260620 14:24:21.929154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 2327
I20260620 14:24:21.929222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:21.929427 392 external_mini_cluster.cc:1428] Setting key 5388bbc3663a862da62307ef0a7e9ea3
I20260620 14:24:21.930549 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:21.925003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43845 (local address 127.0.98.62:42431)
0620 14:24:21.925046 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:21.925048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:21.925122 (+ 74us) server_negotiation.cc:407] Connection header received
0620 14:24:21.925193 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:21.925194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:21.925203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:21.925247 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:21.925561 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.926037 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.926799 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.927008 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.927080 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:21.928015 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:21.928016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:21.928016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:21.928023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:21.929314 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:21.929722 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:21.929723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:21.929724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:21.929779 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:21.929985 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:21.929986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:21.929986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:21.930129 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:21.930202 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:21.930395 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 14:24:21.930463 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:24:21.930569 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:21.924924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42431 (local address 127.0.98.2:43845)
0620 14:24:21.925067 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:21.925078 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:21.925138 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:21.925259 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:21.925260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:21.925430 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:21.925482 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:21.925483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.926134 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.926135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:21.926717 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:21.926719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:21.927090 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:21.927091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:21.927168 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:21.927637 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:21.927642 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:21.927912 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:21.929375 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:21.929376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:21.929376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:21.929641 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:21.929853 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:21.929853 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:21.929854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:21.929914 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:21.930207 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:21.930208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:21.930248 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:24:21.930335 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:24:21.930455 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:21.953446 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42431
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:22.036902 2467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:22.037174 2467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:22.037200 2467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:22.038403 2467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:22.038444 2467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:22.038471 2467 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:24:22.038484 2467 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:24:22.038501 2467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:22.039865 2467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:22.040045 2467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:22.040223 2467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:22.045145 2472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:22.045156 2473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:22.045168 2475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:22.045456 2467 server_base.cc:1034] running on GCE node
I20260620 14:24:22.045651 2467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:22.045881 2467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:22.047029 2467 hybrid_clock.cc:648] HybridClock initialized: now 1781965462047017 us; error 62 us; skew 500 ppm
Jun 20 14:24:22 dist-test-slave-hfc3 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:22.049594 2467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:22.050107 2467 webserver.cc:460] Webserver started at http://127.0.98.3:46771/ using document root <none> and password file <none>
I20260620 14:24:22.050333 2467 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:22.050432 2467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:22.050562 2467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:22.051614 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "b912c67b38b84ea9a35b32682003c36c"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-hfc3"
server_key: "8bda35cdd68dec28cd438df70f802240"
server_key_iv: "203aa0b9f9464a71fcf84ea30a4cd195"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.052008 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b912c67b38b84ea9a35b32682003c36c"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-hfc3"
server_key: "8bda35cdd68dec28cd438df70f802240"
server_key_iv: "203aa0b9f9464a71fcf84ea30a4cd195"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.053483 2467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:22.054227 2482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.054463 2467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:22.054562 2467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "b912c67b38b84ea9a35b32682003c36c"
format_stamp: "Formatted at 2026-06-20 14:24:22 on dist-test-slave-hfc3"
server_key: "8bda35cdd68dec28cd438df70f802240"
server_key_iv: "203aa0b9f9464a71fcf84ea30a4cd195"
server_key_version: "encryptionkey@0"
I20260620 14:24:22.054644 2467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:22.066303 2467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:22.068645 2467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:22.068802 2467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:22.069360 2467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:22.069423 2467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.069454 2467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:22.069471 2467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:24:22 dist-test-slave-hfc3 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:22.076910 2467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43309
I20260620 14:24:22.077003 2597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43309 every 8 connection(s)
I20260620 14:24:22.077376 2467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:22.078325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 2467
I20260620 14:24:22.078480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:22.078691 392 external_mini_cluster.cc:1428] Setting key a1f01fe7fca7c602e769a7dd25aa086a
I20260620 14:24:22.079475 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.069818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55213 (local address 127.0.98.62:42431)
0620 14:24:22.069876 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.069878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.070083 (+ 205us) server_negotiation.cc:407] Connection header received
0620 14:24:22.070714 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.070715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.070728 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.070780 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:22.071415 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.071922 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.072946 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.073155 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.073243 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.076097 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:22.076099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:22.076100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:22.076109 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:22.077973 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.078370 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.078371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.078372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.078435 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.078705 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.078706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.078706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.078863 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:22.078945 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.079278 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.079359 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:24:22.079641 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.069711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42431 (local address 127.0.98.3:55213)
0620 14:24:22.069963 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.069997 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.070595 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.070853 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.070856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.071129 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:22.071312 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.071316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.072019 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.072020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.072830 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.072832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.073252 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.073253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.073376 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.073918 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:22.073927 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:22.075982 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:22.078047 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:22.078051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:22.078053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:22.078260 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:22.078509 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:22.078509 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:22.078510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:22.078631 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:22.078945 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:22.078951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:22.079131 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.079243 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.079412 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":108,"threads_started":1}
I20260620 14:24:22.080098 2598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42431
I20260620 14:24:22.080199 2598 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:22.080368 2598 heartbeater.cc:511] Master 127.0.98.62:42431 requested a full tablet report, sending...
I20260620 14:24:22.080719 2115 ts_manager.cc:194] Registered new tserver with Master: b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309)
I20260620 14:24:22.081163 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55213
I20260620 14:24:22.086428 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.081145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33493 (local address 127.0.98.62:42431)
0620 14:24:22.081186 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.081188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.081455 (+ 267us) server_negotiation.cc:407] Connection header received
0620 14:24:22.081474 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.081474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.081484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.081528 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:22.081880 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.082402 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.083063 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.083242 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.083306 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.084162 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:22.084163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:22.084164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:22.084169 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:22.085390 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.085710 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.085711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.085711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.085756 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.085949 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.085950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.085950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.086050 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:22.086099 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.086205 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.086274 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:24:22.086470 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.081083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42431 (local address 127.0.98.3:33493)
0620 14:24:22.081199 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.081210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.081269 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.081551 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.081552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.081744 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:22.081805 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.081807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.082486 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.082486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.082971 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.082972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.083313 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.083313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.083364 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.083795 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:22.083800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:22.084089 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:22.085475 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:22.085476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:22.085476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:22.085644 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:22.085813 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:22.085813 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:22.085814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:22.085896 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:22.086104 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:22.086104 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:22.086138 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.086189 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.086294 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
I20260620 14:24:22.090690 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:24:22 dist-test-slave-hfc3 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:24:22 dist-test-slave-hfc3 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:22.114200 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.106631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.0.98.62:42431)
0620 14:24:22.106693 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.106695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.106711 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:24:22.106735 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.106736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.106745 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.106797 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:22.107411 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.107911 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.108718 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.108907 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.108977 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.111629 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:22.111631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:22.111631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:22.111638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:22.113166 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.113525 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.113526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.113526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.113580 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.113802 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.113803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.113803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.113903 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:22.113976 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.114081 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.114137 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:24:22.115887 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:22.117404 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:22.124341 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.119861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34503 (local address 127.0.0.1:35330)
0620 14:24:22.119994 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.120036 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.120122 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.120373 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.120376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.120378 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:22.120447 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.120449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.121809 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.121810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.123155 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.123158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.124016 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.124017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.124099 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.124103 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.124151 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.124226 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260620 14:24:22.124375 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.120363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.0.98.3:43309)
0620 14:24:22.120643 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.120647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.120700 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:24:22.120832 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.120835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.120854 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.120938 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:22.121070 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.121804 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.123126 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.123911 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.123991 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.124067 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.124217 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.124258 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":155,"threads_started":1}
I20260620 14:24:22.124620 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.120102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46237 (local address 127.0.0.1:41940)
0620 14:24:22.120226 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.120237 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.120285 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.121037 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.121038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.121040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:22.121110 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.121112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.122391 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.122392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.123410 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.123411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.124393 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.124394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.124472 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.124477 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.124524 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.124561 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260620 14:24:22.124681 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.120265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.0.1:44622)
0620 14:24:22.120639 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.120650 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.120710 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.120947 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.120948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.120951 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:22.121010 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.121011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.121881 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.121882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.123038 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.123041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.124000 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.124001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.124081 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.124096 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.124190 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.124241 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":36,"thread_start_us":63,"threads_started":1}
I20260620 14:24:22.124761 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.119970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.0.98.1:34503)
0620 14:24:22.120151 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.120156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.120166 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:22.120208 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.120210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.120225 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.120296 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:22.120586 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.121719 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.123245 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.123915 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.124299 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.124377 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.124405 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.124645 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260620 14:24:22.124984 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.120472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.0.98.2:46237)
0620 14:24:22.120826 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.120830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.120839 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:24:22.120874 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.120877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.120892 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.120972 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:22.121193 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.122254 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.123498 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.124231 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.124303 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.124358 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.124809 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.124852 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":81,"threads_started":1}
I20260620 14:24:22.125386 2252 tablet_service.cc:1467] Processing CreateTablet for tablet ded8204eef214b639540c8c582115ef8 (DEFAULT_TABLE table=test-table [id=101d60eb725147e0a2b032ba22c89445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:22.125387 2532 tablet_service.cc:1467] Processing CreateTablet for tablet ded8204eef214b639540c8c582115ef8 (DEFAULT_TABLE table=test-table [id=101d60eb725147e0a2b032ba22c89445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:22.125751 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded8204eef214b639540c8c582115ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:22.125763 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded8204eef214b639540c8c582115ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:22.126768 2392 tablet_service.cc:1467] Processing CreateTablet for tablet ded8204eef214b639540c8c582115ef8 (DEFAULT_TABLE table=test-table [id=101d60eb725147e0a2b032ba22c89445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:22.127108 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded8204eef214b639540c8c582115ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:22.128793 2621 tablet_bootstrap.cc:492] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de: Bootstrap starting.
I20260620 14:24:22.129048 2622 tablet_bootstrap.cc:492] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c: Bootstrap starting.
I20260620 14:24:22.129557 2621 tablet_bootstrap.cc:654] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:22.129865 2621 log.cc:824] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:22.129957 2622 tablet_bootstrap.cc:654] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:22.130321 2622 log.cc:824] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:22.130659 2621 tablet_bootstrap.cc:492] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de: No bootstrap required, opened a new log
I20260620 14:24:22.130736 2621 ts_tablet_manager.cc:1397] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:22.130964 2625 tablet_bootstrap.cc:492] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105: Bootstrap starting.
I20260620 14:24:22.131073 2622 tablet_bootstrap.cc:492] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c: No bootstrap required, opened a new log
I20260620 14:24:22.131141 2622 ts_tablet_manager.cc:1397] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:24:22.131560 2625 tablet_bootstrap.cc:654] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:22.131855 2625 log.cc:824] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:22.132543 2625 tablet_bootstrap.cc:492] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105: No bootstrap required, opened a new log
I20260620 14:24:22.132623 2625 ts_tablet_manager.cc:1397] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:22.132859 2621 raft_consensus.cc:348] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.133041 2621 raft_consensus.cc:374] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:22.133080 2621 raft_consensus.cc:729] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8f1efc97f94c7a8777472e8f88a0de, State: Initialized, Role: FOLLOWER
I20260620 14:24:22.133035 2622 raft_consensus.cc:348] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.133209 2622 raft_consensus.cc:374] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:22.133195 2621 consensus_queue.cc:260] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.133261 2622 raft_consensus.cc:729] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b912c67b38b84ea9a35b32682003c36c, State: Initialized, Role: FOLLOWER
I20260620 14:24:22.133389 2622 consensus_queue.cc:260] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.133508 2621 ts_tablet_manager.cc:1428] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:24:22.133620 2318 heartbeater.cc:503] Master 127.0.98.62:42431 was elected leader, sending a full tablet report...
I20260620 14:24:22.133649 2622 ts_tablet_manager.cc:1428] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:24:22.133769 2598 heartbeater.cc:503] Master 127.0.98.62:42431 was elected leader, sending a full tablet report...
I20260620 14:24:22.134709 2625 raft_consensus.cc:348] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.134871 2625 raft_consensus.cc:374] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:22.134927 2625 raft_consensus.cc:729] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec32d73bdb9d430c90b822b147fd1105, State: Initialized, Role: FOLLOWER
I20260620 14:24:22.135041 2625 consensus_queue.cc:260] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.135342 2625 ts_tablet_manager.cc:1428] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:24:22.135403 2458 heartbeater.cc:503] Master 127.0.98.62:42431 was elected leader, sending a full tablet report...
W20260620 14:24:22.170840 2459 tablet.cc:2369] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:22.199800 2629 raft_consensus.cc:482] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:22.199926 2629 raft_consensus.cc:504] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.200342 2629 leader_election.cc:288] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503), b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309)
I20260620 14:24:22.204427 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.200703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38987 (local address 127.0.98.1:34503)
0620 14:24:22.200753 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.200755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.200766 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:24:22.200796 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.200797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.200809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.200864 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:22.201087 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.201904 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.203427 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.203992 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.204080 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.204091 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.204318 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.204358 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:24:22.204427 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.200830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39691 (local address 127.0.98.3:43309)
0620 14:24:22.200903 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.200905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.201058 (+ 153us) server_negotiation.cc:407] Connection header received
0620 14:24:22.201110 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.201111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.201124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.201191 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:22.201316 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.202004 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.203427 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.203996 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.204080 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.204091 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.204318 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.204359 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:24:22.204521 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.200600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34503 (local address 127.0.98.2:38987)
0620 14:24:22.200667 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.200683 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.200742 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.200918 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.200919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.200921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:22.200999 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.201001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.202004 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.202005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.203315 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.203318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.204115 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.204116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.204231 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.204235 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.204299 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.204345 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":28}
I20260620 14:24:22.204454 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.200731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.98.2:39691)
0620 14:24:22.200970 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.200980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.201051 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.201200 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.201201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.201202 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:22.201247 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.201248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.202097 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.202097 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.203316 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.203318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.204115 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.204116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.204232 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.204235 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.204299 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.204345 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":46,"thread_start_us":134,"threads_started":1}
I20260620 14:24:22.204926 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded8204eef214b639540c8c582115ef8" candidate_uuid: "ec32d73bdb9d430c90b822b147fd1105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b912c67b38b84ea9a35b32682003c36c" is_pre_election: true
I20260620 14:24:22.204923 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded8204eef214b639540c8c582115ef8" candidate_uuid: "ec32d73bdb9d430c90b822b147fd1105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" is_pre_election: true
I20260620 14:24:22.205087 2552 raft_consensus.cc:2454] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec32d73bdb9d430c90b822b147fd1105 in term 0.
I20260620 14:24:22.205106 2272 raft_consensus.cc:2454] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec32d73bdb9d430c90b822b147fd1105 in term 0.
I20260620 14:24:22.205341 2344 leader_election.cc:302] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b912c67b38b84ea9a35b32682003c36c, ec32d73bdb9d430c90b822b147fd1105; no voters:
I20260620 14:24:22.205509 2629 raft_consensus.cc:2781] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:22.205583 2629 raft_consensus.cc:482] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:22.205601 2629 raft_consensus.cc:3037] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:22.206702 2629 raft_consensus.cc:504] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.206840 2629 leader_election.cc:288] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [CANDIDATE]: Term 1 election: Requested vote from peers 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503), b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309)
I20260620 14:24:22.207065 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded8204eef214b639540c8c582115ef8" candidate_uuid: "ec32d73bdb9d430c90b822b147fd1105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8f1efc97f94c7a8777472e8f88a0de"
I20260620 14:24:22.207074 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded8204eef214b639540c8c582115ef8" candidate_uuid: "ec32d73bdb9d430c90b822b147fd1105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b912c67b38b84ea9a35b32682003c36c"
I20260620 14:24:22.207146 2272 raft_consensus.cc:3037] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:22.207178 2552 raft_consensus.cc:3037] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:22.207858 2272 raft_consensus.cc:2454] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec32d73bdb9d430c90b822b147fd1105 in term 1.
I20260620 14:24:22.207927 2552 raft_consensus.cc:2454] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec32d73bdb9d430c90b822b147fd1105 in term 1.
I20260620 14:24:22.208060 2344 leader_election.cc:302] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f8f1efc97f94c7a8777472e8f88a0de, ec32d73bdb9d430c90b822b147fd1105; no voters:
I20260620 14:24:22.208185 2629 raft_consensus.cc:2781] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:22.208374 2629 raft_consensus.cc:686] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 1 LEADER]: Becoming Leader. State: Replica: ec32d73bdb9d430c90b822b147fd1105, State: Running, Role: LEADER
I20260620 14:24:22.208491 2629 consensus_queue.cc:237] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.209378 2115 catalog_manager.cc:5495] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec32d73bdb9d430c90b822b147fd1105 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ec32d73bdb9d430c90b822b147fd1105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:24:22 dist-test-slave-hfc3 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:22.233734 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.227022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.0.98.62:42431)
0620 14:24:22.227084 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.227087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.227191 (+ 104us) server_negotiation.cc:407] Connection header received
0620 14:24:22.227289 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.227290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.227306 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.227368 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:22.227841 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.228336 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.229036 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.229214 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.229298 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.231364 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:22.231366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:22.231366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:22.231373 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:22.232568 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.232946 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.232947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.232948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.233002 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:22.233206 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:22.233207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:22.233207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:22.233313 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:22.233385 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.233539 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.233624 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:24:22.233820 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.226968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42431 (local address 127.0.0.1:56560)
0620 14:24:22.227123 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.227132 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.227209 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.227373 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.227374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.227709 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:22.227751 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.227754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.228424 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.228425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.228946 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.228948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.229298 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.229299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.229355 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.229913 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:22.229917 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:22.231273 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:22.232662 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:22.232669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:22.232672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:22.232860 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:22.233083 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:22.233084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:22.233085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:22.233144 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:22.233377 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:22.233384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:22.233431 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.233523 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.233698 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":78,"thread_start_us":85,"threads_started":1}
I20260620 14:24:22.235878 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56560:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:24:22.236102 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:22.237576 2252 tablet_service.cc:1467] Processing CreateTablet for tablet cdd34f7b11014cd6a6d89f048d627bc4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4d1e069516dd488995272d3502061e49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:24:22.237638 2392 tablet_service.cc:1467] Processing CreateTablet for tablet cdd34f7b11014cd6a6d89f048d627bc4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4d1e069516dd488995272d3502061e49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:24:22.237648 2532 tablet_service.cc:1467] Processing CreateTablet for tablet cdd34f7b11014cd6a6d89f048d627bc4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4d1e069516dd488995272d3502061e49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:24:22.237772 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd34f7b11014cd6a6d89f048d627bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:22.237772 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd34f7b11014cd6a6d89f048d627bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:22.237771 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd34f7b11014cd6a6d89f048d627bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:22.239058 2625 tablet_bootstrap.cc:492] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105: Bootstrap starting.
I20260620 14:24:22.239297 2622 tablet_bootstrap.cc:492] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: Bootstrap starting.
I20260620 14:24:22.239529 2625 tablet_bootstrap.cc:654] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:22.239825 2622 tablet_bootstrap.cc:654] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:22.240181 2625 tablet_bootstrap.cc:492] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105: No bootstrap required, opened a new log
I20260620 14:24:22.240206 2621 tablet_bootstrap.cc:492] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de: Bootstrap starting.
I20260620 14:24:22.240249 2625 ts_tablet_manager.cc:1397] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:22.240404 2625 raft_consensus.cc:348] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.240494 2625 raft_consensus.cc:374] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:22.240516 2625 raft_consensus.cc:729] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec32d73bdb9d430c90b822b147fd1105, State: Initialized, Role: FOLLOWER
I20260620 14:24:22.240571 2625 consensus_queue.cc:260] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.240664 2625 ts_tablet_manager.cc:1428] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:22.240701 2629 tablet_replica.cc:439] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } }
I20260620 14:24:22.240772 2629 tablet_replica.cc:442] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:22.240839 2621 tablet_bootstrap.cc:654] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:22.241276 2622 tablet_bootstrap.cc:492] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: No bootstrap required, opened a new log
I20260620 14:24:22.241334 2622 ts_tablet_manager.cc:1397] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:24:22.241467 2621 tablet_bootstrap.cc:492] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de: No bootstrap required, opened a new log
I20260620 14:24:22.241477 2622 raft_consensus.cc:348] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.241539 2621 ts_tablet_manager.cc:1397] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:22.241551 2622 raft_consensus.cc:374] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:22.241580 2622 raft_consensus.cc:729] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b912c67b38b84ea9a35b32682003c36c, State: Initialized, Role: FOLLOWER
I20260620 14:24:22.241619 2622 consensus_queue.cc:260] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.241717 2622 ts_tablet_manager.cc:1428] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.241698 2621 raft_consensus.cc:348] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.241760 2621 raft_consensus.cc:374] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:22.241778 2621 raft_consensus.cc:729] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8f1efc97f94c7a8777472e8f88a0de, State: Initialized, Role: FOLLOWER
I20260620 14:24:22.241801 2628 tablet_replica.cc:439] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } }
I20260620 14:24:22.241823 2621 consensus_queue.cc:260] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.241875 2628 tablet_replica.cc:442] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:22.241925 2621 ts_tablet_manager.cc:1428] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.242303 2627 tablet_replica.cc:439] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } }
I20260620 14:24:22.242415 2627 tablet_replica.cc:442] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:22.258121 2628 raft_consensus.cc:482] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:22.258204 2628 raft_consensus.cc:504] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.258652 2628 leader_election.cc:288] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503), ec32d73bdb9d430c90b822b147fd1105 (127.0.98.2:46237)
I20260620 14:24:22.261493 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.258931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52575 (local address 127.0.98.1:34503)
0620 14:24:22.258997 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.258999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.259009 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:22.259037 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.259037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.259047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.259113 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:22.259303 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.259821 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.260756 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.261130 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.261223 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.261234 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.261398 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.261437 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:24:22.261492 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.258832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34503 (local address 127.0.98.3:52575)
0620 14:24:22.258886 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.258898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.258941 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.259137 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.259138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.259140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:22.259213 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.259214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.259909 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.259910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.260656 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.260658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.261225 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.261226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.261310 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.261314 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.261377 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.261421 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:24:22.261826 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.259037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48711 (local address 127.0.98.2:46237)
0620 14:24:22.259126 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.259129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.259193 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:24:22.259277 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.259279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.259290 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.259339 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:22.259546 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.260026 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.260952 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.261307 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.261409 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.261419 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.261717 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.261760 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:24:22.261885 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd34f7b11014cd6a6d89f048d627bc4" candidate_uuid: "b912c67b38b84ea9a35b32682003c36c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" is_pre_election: true
I20260620 14:24:22.261982 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.258956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46237 (local address 127.0.98.3:48711)
0620 14:24:22.259126 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.259137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.259219 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.259382 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.259383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.259385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:22.259459 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.259461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.260112 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.260113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.260833 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.260835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.261438 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.261439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.261513 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.261516 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.261585 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.261898 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":124,"threads_started":1}
I20260620 14:24:22.261987 2272 raft_consensus.cc:2454] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b912c67b38b84ea9a35b32682003c36c in term 0.
I20260620 14:24:22.262207 2484 leader_election.cc:302] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f8f1efc97f94c7a8777472e8f88a0de, b912c67b38b84ea9a35b32682003c36c; no voters:
I20260620 14:24:22.262369 2628 raft_consensus.cc:2781] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:22.262305 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd34f7b11014cd6a6d89f048d627bc4" candidate_uuid: "b912c67b38b84ea9a35b32682003c36c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec32d73bdb9d430c90b822b147fd1105" is_pre_election: true
I20260620 14:24:22.262413 2628 raft_consensus.cc:482] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:22.262440 2628 raft_consensus.cc:3037] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:22.262451 2412 raft_consensus.cc:2454] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b912c67b38b84ea9a35b32682003c36c in term 0.
I20260620 14:24:22.263298 2628 raft_consensus.cc:504] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.263427 2628 leader_election.cc:288] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [CANDIDATE]: Term 1 election: Requested vote from peers 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503), ec32d73bdb9d430c90b822b147fd1105 (127.0.98.2:46237)
I20260620 14:24:22.263602 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd34f7b11014cd6a6d89f048d627bc4" candidate_uuid: "b912c67b38b84ea9a35b32682003c36c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8f1efc97f94c7a8777472e8f88a0de"
I20260620 14:24:22.263684 2272 raft_consensus.cc:3037] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:22.263621 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd34f7b11014cd6a6d89f048d627bc4" candidate_uuid: "b912c67b38b84ea9a35b32682003c36c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec32d73bdb9d430c90b822b147fd1105"
I20260620 14:24:22.263744 2412 raft_consensus.cc:3037] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:22.264596 2272 raft_consensus.cc:2454] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b912c67b38b84ea9a35b32682003c36c in term 1.
I20260620 14:24:22.264657 2412 raft_consensus.cc:2454] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b912c67b38b84ea9a35b32682003c36c in term 1.
W20260620 14:24:22.264736 2319 tablet.cc:2369] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:22.264808 2484 leader_election.cc:302] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b912c67b38b84ea9a35b32682003c36c, ec32d73bdb9d430c90b822b147fd1105; no voters:
I20260620 14:24:22.264925 2628 raft_consensus.cc:2781] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:22.265110 2628 raft_consensus.cc:686] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [term 1 LEADER]: Becoming Leader. State: Replica: b912c67b38b84ea9a35b32682003c36c, State: Running, Role: LEADER
I20260620 14:24:22.265221 2628 consensus_queue.cc:237] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.265949 2643 tablet_replica.cc:439] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: TxnStatusTablet state changed. Reason: New leader b912c67b38b84ea9a35b32682003c36c. Latest consensus state: current_term: 1 leader_uuid: "b912c67b38b84ea9a35b32682003c36c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } }
I20260620 14:24:22.266048 2643 tablet_replica.cc:442] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:24:22.266142 2115 catalog_manager.cc:5495] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c reported cstate change: term changed from 0 to 1, leader changed from <none> to b912c67b38b84ea9a35b32682003c36c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b912c67b38b84ea9a35b32682003c36c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } health_report { overall_health: HEALTHY } } }
I20260620 14:24:22.266260 2645 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:24:22.266392 2645 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:24:22.278056 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.275569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.0.98.3:43309)
0620 14:24:22.275617 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.275618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.275629 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:24:22.275711 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.275712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.275730 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.275813 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:22.275940 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.276409 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.277217 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.277426 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.277519 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.277715 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:22.277818 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:22.277907 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.277949 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.277970 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:24:22.278090 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.275529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.0.1:44636)
0620 14:24:22.275588 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.275599 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.275650 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.275812 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.275813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.275814 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:24:22.275855 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.275857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.276494 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.276495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.277080 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.277082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.277516 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.277517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.277589 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.277604 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:24:22.277887 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:24:22.277897 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.277947 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.277999 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 14:24:22.279994 2272 raft_consensus.cc:1261] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Refusing update from remote peer b912c67b38b84ea9a35b32682003c36c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:22.280278 2628 consensus_queue.cc:1035] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:22.280370 2412 raft_consensus.cc:1261] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Refusing update from remote peer b912c67b38b84ea9a35b32682003c36c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:22.280622 2628 consensus_queue.cc:1035] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:22.281668 2627 tablet_replica.cc:439] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de: TxnStatusTablet state changed. Reason: New leader b912c67b38b84ea9a35b32682003c36c. Latest consensus state: current_term: 1 leader_uuid: "b912c67b38b84ea9a35b32682003c36c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } }
I20260620 14:24:22.281761 2627 tablet_replica.cc:442] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:22.282752 2627 tablet_replica.cc:439] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b912c67b38b84ea9a35b32682003c36c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } }
I20260620 14:24:22.282774 2643 tablet_replica.cc:439] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b912c67b38b84ea9a35b32682003c36c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } }
I20260620 14:24:22.282835 2627 tablet_replica.cc:442] T cdd34f7b11014cd6a6d89f048d627bc4 P 3f8f1efc97f94c7a8777472e8f88a0de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:22.282851 2643 tablet_replica.cc:442] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:24:22.283284 2629 tablet_replica.cc:439] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105: TxnStatusTablet state changed. Reason: New leader b912c67b38b84ea9a35b32682003c36c. Latest consensus state: current_term: 1 leader_uuid: "b912c67b38b84ea9a35b32682003c36c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } }
I20260620 14:24:22.283375 2629 tablet_replica.cc:442] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:22.283950 2628 tablet_replica.cc:439] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b912c67b38b84ea9a35b32682003c36c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } }
I20260620 14:24:22.284036 2628 tablet_replica.cc:442] T cdd34f7b11014cd6a6d89f048d627bc4 P b912c67b38b84ea9a35b32682003c36c: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:24:22.284439 2629 tablet_replica.cc:439] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b912c67b38b84ea9a35b32682003c36c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } }
I20260620 14:24:22.284528 2629 tablet_replica.cc:442] T cdd34f7b11014cd6a6d89f048d627bc4 P ec32d73bdb9d430c90b822b147fd1105: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:22.283890 2649 mvcc.cc:205] Tried to move back new op lower bound from 7298930533496295424 to 7298930533439057920. Current Snapshot: MvccSnapshot[applied={T|T < 7298930533496295424}]
I20260620 14:24:22.289311 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.286901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.0.98.2:46237)
0620 14:24:22.286938 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.286940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.286995 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:24:22.287072 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.287073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.287089 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.287157 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:22.287295 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.287762 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.288523 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.288731 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.288812 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.288988 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:22.289086 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:22.289166 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.289217 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.289238 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:24:22.293964 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.291778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42683 (local address 127.0.98.3:43309)
0620 14:24:22.291840 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.291842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.291850 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:22.291871 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.291872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.291882 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.291949 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:22.292089 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.292533 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.293268 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.293456 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.293528 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.293725 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:22.293759 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:22.293821 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.293880 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.293899 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:24:22.293980 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.291684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.98.2:42683)
0620 14:24:22.291745 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.291755 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.291811 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.291951 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.291951 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.291952 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:24:22.292004 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.292005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.292626 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.292627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.293185 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.293186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.293543 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.293544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.293611 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.293624 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:24:22.293822 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:24:22.293824 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.293875 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.293915 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:24:22.296422 2272 raft_consensus.cc:1261] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Refusing update from remote peer ec32d73bdb9d430c90b822b147fd1105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:22.296515 2552 raft_consensus.cc:1261] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Refusing update from remote peer ec32d73bdb9d430c90b822b147fd1105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:22.296707 2629 consensus_queue.cc:1035] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:22.296842 2631 consensus_queue.cc:1035] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:22.298254 2653 mvcc.cc:205] Tried to move back new op lower bound from 7298930533564059648 to 7298930533206503424. Current Snapshot: MvccSnapshot[applied={T|T < 7298930533564059648}]
I20260620 14:24:22.298890 2648 mvcc.cc:205] Tried to move back new op lower bound from 7298930533564059648 to 7298930533206503424. Current Snapshot: MvccSnapshot[applied={T|T < 7298930533564059648}]
I20260620 14:24:22.305509 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.302915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46237 (local address 127.0.98.3:59955)
0620 14:24:22.302988 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:22.303000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:22.303047 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:22.303220 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:22.303221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:22.303222 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:24:22.303276 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.303277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.303968 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.303970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.304562 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:22.304564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.304979 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.304980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:22.305062 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.305077 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:24:22.305331 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:24:22.305334 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:22.305387 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:24:22.305432 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:24:22.305693 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:22.302986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59955 (local address 127.0.98.2:46237)
0620 14:24:22.303119 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:22.303121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:22.303128 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:24:22.303148 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:22.303148 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:22.303159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:22.303201 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:22.303353 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.303885 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.304660 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:22.304887 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:22.304946 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:22.305232 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:22.305258 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:22.305328 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:22.305611 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 14:24:22.305630 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 14:24:22.312525 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:22.312642 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:22.313338 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735: Sending DeleteTablet for 3 replicas of tablet ded8204eef214b639540c8c582115ef8
I20260620 14:24:22.313755 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet ded8204eef214b639540c8c582115ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35330
I20260620 14:24:22.313984 2667 tablet_replica.cc:330] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de: stopping tablet replica
I20260620 14:24:22.313968 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet ded8204eef214b639540c8c582115ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41940
I20260620 14:24:22.314184 2667 raft_consensus.cc:2229] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:22.314186 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet ded8204eef214b639540c8c582115ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44622
I20260620 14:24:22.314316 2667 raft_consensus.cc:2258] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:22.314396 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:22.314555 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:22.315688 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1659bf5d9449d592d082d26da5ba34 (DEFAULT_TABLE table=test-table [id=df4c8ac6e80445a29f7e16f26b4cb346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:22.315840 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c1659bf5d9449d592d082d26da5ba34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:22.316843 2625 tablet_bootstrap.cc:492] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105: Bootstrap starting.
I20260620 14:24:22.317090 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1659bf5d9449d592d082d26da5ba34 (DEFAULT_TABLE table=test-table [id=df4c8ac6e80445a29f7e16f26b4cb346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:22.317199 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c1659bf5d9449d592d082d26da5ba34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:22.317276 2625 tablet_bootstrap.cc:654] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:22.318292 2621 tablet_bootstrap.cc:492] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de: Bootstrap starting.
I20260620 14:24:22.318564 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1659bf5d9449d592d082d26da5ba34 (DEFAULT_TABLE table=test-table [id=df4c8ac6e80445a29f7e16f26b4cb346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:22.318663 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c1659bf5d9449d592d082d26da5ba34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:22.318732 2621 tablet_bootstrap.cc:654] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:22.319380 2621 tablet_bootstrap.cc:492] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de: No bootstrap required, opened a new log
I20260620 14:24:22.319433 2621 ts_tablet_manager.cc:1397] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:22.319574 2621 raft_consensus.cc:348] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.319633 2621 raft_consensus.cc:374] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:22.319653 2621 raft_consensus.cc:729] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8f1efc97f94c7a8777472e8f88a0de, State: Initialized, Role: FOLLOWER
I20260620 14:24:22.319695 2621 consensus_queue.cc:260] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.319769 2621 ts_tablet_manager.cc:1428] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.320070 2622 tablet_bootstrap.cc:492] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c: Bootstrap starting.
I20260620 14:24:22.320500 2622 tablet_bootstrap.cc:654] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:22.320503 2667 ts_tablet_manager.cc:1905] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:22.321156 2622 tablet_bootstrap.cc:492] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c: No bootstrap required, opened a new log
I20260620 14:24:22.321209 2622 ts_tablet_manager.cc:1397] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:22.321341 2622 raft_consensus.cc:348] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.321409 2622 raft_consensus.cc:374] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:22.321430 2622 raft_consensus.cc:729] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b912c67b38b84ea9a35b32682003c36c, State: Initialized, Role: FOLLOWER
I20260620 14:24:22.321471 2622 consensus_queue.cc:260] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.321556 2622 ts_tablet_manager.cc:1428] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:22.322312 2667 ts_tablet_manager.cc:1918] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:22.322417 2667 log.cc:1196] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/wal/wals/ded8204eef214b639540c8c582115ef8
I20260620 14:24:22.322553 2669 tablet_replica.cc:330] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c: stopping tablet replica
I20260620 14:24:22.322711 2667 ts_tablet_manager.cc:1939] T ded8204eef214b639540c8c582115ef8 P 3f8f1efc97f94c7a8777472e8f88a0de: Deleting consensus metadata
I20260620 14:24:22.322872 2669 raft_consensus.cc:2229] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:22.323102 2669 raft_consensus.cc:2258] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:22.323103 2100 catalog_manager.cc:4841] TS 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503): tablet ded8204eef214b639540c8c582115ef8 (table test-table [id=101d60eb725147e0a2b032ba22c89445]) successfully deleted
I20260620 14:24:22.323348 2625 tablet_bootstrap.cc:492] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105: No bootstrap required, opened a new log
I20260620 14:24:22.323433 2625 ts_tablet_manager.cc:1397] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 14:24:22.323589 2625 raft_consensus.cc:348] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.323664 2625 raft_consensus.cc:374] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:22.323693 2625 raft_consensus.cc:729] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec32d73bdb9d430c90b822b147fd1105, State: Initialized, Role: FOLLOWER
I20260620 14:24:22.323733 2625 consensus_queue.cc:260] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.323825 2625 ts_tablet_manager.cc:1428] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:22.323925 2668 tablet_replica.cc:330] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105: stopping tablet replica
I20260620 14:24:22.324028 2668 raft_consensus.cc:2229] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:22.324126 2668 raft_consensus.cc:2258] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:22.324350 2669 ts_tablet_manager.cc:1905] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:22.324579 2668 ts_tablet_manager.cc:1905] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:22.325723 2669 ts_tablet_manager.cc:1918] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:22.325810 2669 log.cc:1196] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/ded8204eef214b639540c8c582115ef8
I20260620 14:24:22.326066 2669 ts_tablet_manager.cc:1939] T ded8204eef214b639540c8c582115ef8 P b912c67b38b84ea9a35b32682003c36c: Deleting consensus metadata
I20260620 14:24:22.326200 2668 ts_tablet_manager.cc:1918] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:22.326274 2668 log.cc:1196] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/ded8204eef214b639540c8c582115ef8
I20260620 14:24:22.326490 2100 catalog_manager.cc:4841] TS b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309): tablet ded8204eef214b639540c8c582115ef8 (table test-table [id=101d60eb725147e0a2b032ba22c89445]) successfully deleted
I20260620 14:24:22.326565 2668 ts_tablet_manager.cc:1939] T ded8204eef214b639540c8c582115ef8 P ec32d73bdb9d430c90b822b147fd1105: Deleting consensus metadata
I20260620 14:24:22.326910 2100 catalog_manager.cc:4841] TS ec32d73bdb9d430c90b822b147fd1105 (127.0.98.2:46237): tablet ded8204eef214b639540c8c582115ef8 (table test-table [id=101d60eb725147e0a2b032ba22c89445]) successfully deleted
W20260620 14:24:22.328117 2599 tablet.cc:2369] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:22.682037 2665 raft_consensus.cc:482] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:22.682134 2665 raft_consensus.cc:504] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.682497 2665 leader_election.cc:288] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503), b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309)
I20260620 14:24:22.682775 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1659bf5d9449d592d082d26da5ba34" candidate_uuid: "ec32d73bdb9d430c90b822b147fd1105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b912c67b38b84ea9a35b32682003c36c" is_pre_election: true
I20260620 14:24:22.682866 2552 raft_consensus.cc:2454] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec32d73bdb9d430c90b822b147fd1105 in term 0.
I20260620 14:24:22.682801 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1659bf5d9449d592d082d26da5ba34" candidate_uuid: "ec32d73bdb9d430c90b822b147fd1105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" is_pre_election: true
I20260620 14:24:22.682926 2272 raft_consensus.cc:2454] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec32d73bdb9d430c90b822b147fd1105 in term 0.
I20260620 14:24:22.683053 2344 leader_election.cc:302] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b912c67b38b84ea9a35b32682003c36c, ec32d73bdb9d430c90b822b147fd1105; no voters:
I20260620 14:24:22.683164 2665 raft_consensus.cc:2781] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:22.683205 2665 raft_consensus.cc:482] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:22.683229 2665 raft_consensus.cc:3037] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:22.683967 2665 raft_consensus.cc:504] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.684111 2665 leader_election.cc:288] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [CANDIDATE]: Term 1 election: Requested vote from peers 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503), b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309)
I20260620 14:24:22.684319 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1659bf5d9449d592d082d26da5ba34" candidate_uuid: "ec32d73bdb9d430c90b822b147fd1105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8f1efc97f94c7a8777472e8f88a0de"
I20260620 14:24:22.684319 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1659bf5d9449d592d082d26da5ba34" candidate_uuid: "ec32d73bdb9d430c90b822b147fd1105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b912c67b38b84ea9a35b32682003c36c"
I20260620 14:24:22.684401 2552 raft_consensus.cc:3037] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:22.684401 2272 raft_consensus.cc:3037] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:22.685156 2552 raft_consensus.cc:2454] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec32d73bdb9d430c90b822b147fd1105 in term 1.
I20260620 14:24:22.685155 2272 raft_consensus.cc:2454] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec32d73bdb9d430c90b822b147fd1105 in term 1.
I20260620 14:24:22.685343 2344 leader_election.cc:302] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b912c67b38b84ea9a35b32682003c36c, ec32d73bdb9d430c90b822b147fd1105; no voters:
I20260620 14:24:22.685465 2665 raft_consensus.cc:2781] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:22.685534 2665 raft_consensus.cc:686] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 1 LEADER]: Becoming Leader. State: Replica: ec32d73bdb9d430c90b822b147fd1105, State: Running, Role: LEADER
I20260620 14:24:22.685598 2665 consensus_queue.cc:237] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:22.686184 2114 catalog_manager.cc:5495] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec32d73bdb9d430c90b822b147fd1105 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ec32d73bdb9d430c90b822b147fd1105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } health_report { overall_health: UNKNOWN } } }
I20260620 14:24:22.763540 2552 raft_consensus.cc:1261] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Refusing update from remote peer ec32d73bdb9d430c90b822b147fd1105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:22.763576 2272 raft_consensus.cc:1261] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Refusing update from remote peer ec32d73bdb9d430c90b822b147fd1105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:22.763777 2665 consensus_queue.cc:1035] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:22.763901 2629 consensus_queue.cc:1035] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:24:22.768585 2653 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:24:24.285835 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:24.285934 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:24.286823 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735: Sending DeleteTablet for 3 replicas of tablet 7c1659bf5d9449d592d082d26da5ba34
I20260620 14:24:24.287170 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c1659bf5d9449d592d082d26da5ba34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35330
I20260620 14:24:24.287190 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c1659bf5d9449d592d082d26da5ba34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44622
I20260620 14:24:24.287184 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c1659bf5d9449d592d082d26da5ba34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41940
I20260620 14:24:24.289041 2698 tablet_replica.cc:330] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105: stopping tablet replica
I20260620 14:24:24.289121 2698 raft_consensus.cc:2229] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:24.289189 2698 raft_consensus.cc:2258] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:24.289302 2699 tablet_replica.cc:330] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c: stopping tablet replica
I20260620 14:24:24.289374 2699 raft_consensus.cc:2229] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:24.289415 2699 raft_consensus.cc:2258] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:24.289529 2700 tablet_replica.cc:330] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de: stopping tablet replica
I20260620 14:24:24.289599 2700 raft_consensus.cc:2229] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:24.289644 2700 raft_consensus.cc:2258] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:24.289904 2700 ts_tablet_manager.cc:1905] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:24.289908 2698 ts_tablet_manager.cc:1905] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:24.290517 2699 ts_tablet_manager.cc:1905] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:24.291965 2700 ts_tablet_manager.cc:1918] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:24:24.292045 2700 log.cc:1196] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/wal/wals/7c1659bf5d9449d592d082d26da5ba34
I20260620 14:24:24.292203 2698 ts_tablet_manager.cc:1918] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:24:24.292315 2700 ts_tablet_manager.cc:1939] T 7c1659bf5d9449d592d082d26da5ba34 P 3f8f1efc97f94c7a8777472e8f88a0de: Deleting consensus metadata
I20260620 14:24:24.292356 2699 ts_tablet_manager.cc:1918] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:24:24.292409 2699 log.cc:1196] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/7c1659bf5d9449d592d082d26da5ba34
I20260620 14:24:24.292352 2698 log.cc:1196] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/7c1659bf5d9449d592d082d26da5ba34
I20260620 14:24:24.292683 2699 ts_tablet_manager.cc:1939] T 7c1659bf5d9449d592d082d26da5ba34 P b912c67b38b84ea9a35b32682003c36c: Deleting consensus metadata
I20260620 14:24:24.292824 2100 catalog_manager.cc:4841] TS 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503): tablet 7c1659bf5d9449d592d082d26da5ba34 (table test-table [id=df4c8ac6e80445a29f7e16f26b4cb346]) successfully deleted
I20260620 14:24:24.292935 2698 ts_tablet_manager.cc:1939] T 7c1659bf5d9449d592d082d26da5ba34 P ec32d73bdb9d430c90b822b147fd1105: Deleting consensus metadata
I20260620 14:24:24.293040 2100 catalog_manager.cc:4841] TS b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309): tablet 7c1659bf5d9449d592d082d26da5ba34 (table test-table [id=df4c8ac6e80445a29f7e16f26b4cb346]) successfully deleted
I20260620 14:24:24.293239 2100 catalog_manager.cc:4841] TS ec32d73bdb9d430c90b822b147fd1105 (127.0.98.2:46237): tablet 7c1659bf5d9449d592d082d26da5ba34 (table test-table [id=df4c8ac6e80445a29f7e16f26b4cb346]) successfully deleted
Jun 20 14:24:24 dist-test-slave-hfc3 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965464, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:24:24 dist-test-slave-hfc3 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965464, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:24.312479 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.304925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.0.98.62:42431)
0620 14:24:24.305071 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.305073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.305084 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:24:24.305115 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.305116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.305128 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.305211 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:24.305802 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.306320 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.307139 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.307353 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.307434 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.309627 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:24.309629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:24.309629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:24.309637 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:24.311227 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:24.311631 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:24.311632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:24.311632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:24.311705 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:24.311925 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:24.311926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:24.311926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:24.312042 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:24.312143 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.312275 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.312363 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260620 14:24:24.314117 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:24.314304 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:24.315696 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 7abd745b7128493380b2e6a7c9d80334 (DEFAULT_TABLE table=test-table [id=ef957346fc0f41b3b2852dc77b15628b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:24.315701 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 7abd745b7128493380b2e6a7c9d80334 (DEFAULT_TABLE table=test-table [id=ef957346fc0f41b3b2852dc77b15628b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:24.315723 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 7abd745b7128493380b2e6a7c9d80334 (DEFAULT_TABLE table=test-table [id=ef957346fc0f41b3b2852dc77b15628b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:24.315842 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abd745b7128493380b2e6a7c9d80334. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:24.315842 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abd745b7128493380b2e6a7c9d80334. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:24.315860 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abd745b7128493380b2e6a7c9d80334. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:24.317234 2710 tablet_bootstrap.cc:492] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c: Bootstrap starting.
I20260620 14:24:24.317256 2711 tablet_bootstrap.cc:492] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105: Bootstrap starting.
I20260620 14:24:24.317432 2709 tablet_bootstrap.cc:492] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de: Bootstrap starting.
I20260620 14:24:24.317658 2711 tablet_bootstrap.cc:654] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:24.317658 2710 tablet_bootstrap.cc:654] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:24.317800 2709 tablet_bootstrap.cc:654] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:24.318423 2711 tablet_bootstrap.cc:492] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105: No bootstrap required, opened a new log
I20260620 14:24:24.318480 2711 ts_tablet_manager.cc:1397] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:24.318495 2710 tablet_bootstrap.cc:492] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c: No bootstrap required, opened a new log
I20260620 14:24:24.318543 2710 ts_tablet_manager.cc:1397] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:24.318601 2711 raft_consensus.cc:348] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.318668 2709 tablet_bootstrap.cc:492] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de: No bootstrap required, opened a new log
I20260620 14:24:24.318734 2709 ts_tablet_manager.cc:1397] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:24.318667 2710 raft_consensus.cc:348] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.318682 2711 raft_consensus.cc:374] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:24.318773 2711 raft_consensus.cc:729] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec32d73bdb9d430c90b822b147fd1105, State: Initialized, Role: FOLLOWER
I20260620 14:24:24.318768 2710 raft_consensus.cc:374] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:24.318799 2710 raft_consensus.cc:729] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b912c67b38b84ea9a35b32682003c36c, State: Initialized, Role: FOLLOWER
I20260620 14:24:24.318826 2711 consensus_queue.cc:260] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.318852 2710 consensus_queue.cc:260] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.318883 2709 raft_consensus.cc:348] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.318944 2709 raft_consensus.cc:374] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:24.318944 2710 ts_tablet_manager.cc:1428] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:24.318969 2709 raft_consensus.cc:729] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8f1efc97f94c7a8777472e8f88a0de, State: Initialized, Role: FOLLOWER
I20260620 14:24:24.319032 2709 consensus_queue.cc:260] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.319056 2711 ts_tablet_manager.cc:1428] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:24.319211 2709 ts_tablet_manager.cc:1428] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:24.579947 2631 raft_consensus.cc:482] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:24.580075 2631 raft_consensus.cc:504] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.580291 2631 leader_election.cc:288] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503), b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309)
I20260620 14:24:24.580601 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abd745b7128493380b2e6a7c9d80334" candidate_uuid: "ec32d73bdb9d430c90b822b147fd1105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b912c67b38b84ea9a35b32682003c36c" is_pre_election: true
I20260620 14:24:24.580602 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abd745b7128493380b2e6a7c9d80334" candidate_uuid: "ec32d73bdb9d430c90b822b147fd1105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" is_pre_election: true
I20260620 14:24:24.580717 2552 raft_consensus.cc:2454] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec32d73bdb9d430c90b822b147fd1105 in term 0.
I20260620 14:24:24.580724 2271 raft_consensus.cc:2454] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec32d73bdb9d430c90b822b147fd1105 in term 0.
I20260620 14:24:24.580924 2344 leader_election.cc:302] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b912c67b38b84ea9a35b32682003c36c, ec32d73bdb9d430c90b822b147fd1105; no voters:
I20260620 14:24:24.581050 2631 raft_consensus.cc:2781] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:24.581111 2631 raft_consensus.cc:482] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:24.581144 2631 raft_consensus.cc:3037] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:24.581864 2631 raft_consensus.cc:504] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.581998 2631 leader_election.cc:288] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [CANDIDATE]: Term 1 election: Requested vote from peers 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503), b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309)
I20260620 14:24:24.582192 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abd745b7128493380b2e6a7c9d80334" candidate_uuid: "ec32d73bdb9d430c90b822b147fd1105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b912c67b38b84ea9a35b32682003c36c"
I20260620 14:24:24.582188 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abd745b7128493380b2e6a7c9d80334" candidate_uuid: "ec32d73bdb9d430c90b822b147fd1105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8f1efc97f94c7a8777472e8f88a0de"
I20260620 14:24:24.582273 2552 raft_consensus.cc:3037] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:24.582273 2271 raft_consensus.cc:3037] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:24.583129 2552 raft_consensus.cc:2454] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec32d73bdb9d430c90b822b147fd1105 in term 1.
I20260620 14:24:24.583129 2271 raft_consensus.cc:2454] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec32d73bdb9d430c90b822b147fd1105 in term 1.
I20260620 14:24:24.583346 2344 leader_election.cc:302] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b912c67b38b84ea9a35b32682003c36c, ec32d73bdb9d430c90b822b147fd1105; no voters:
I20260620 14:24:24.583437 2631 raft_consensus.cc:2781] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:24.583508 2631 raft_consensus.cc:686] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 1 LEADER]: Becoming Leader. State: Replica: ec32d73bdb9d430c90b822b147fd1105, State: Running, Role: LEADER
I20260620 14:24:24.583568 2631 consensus_queue.cc:237] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.584187 2115 catalog_manager.cc:5495] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec32d73bdb9d430c90b822b147fd1105 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ec32d73bdb9d430c90b822b147fd1105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } health_report { overall_health: UNKNOWN } } }
I20260620 14:24:24.604105 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.601361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.0.98.2:46237)
0620 14:24:24.601555 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.601557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.601566 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:24:24.601585 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.601585 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.601595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.601662 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:24.601859 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.602573 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.603365 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.603631 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.603688 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.603846 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:24.603874 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:24.603913 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.603963 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.603981 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":115,"threads_started":1}
I20260620 14:24:24.606514 2271 raft_consensus.cc:1261] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Refusing update from remote peer ec32d73bdb9d430c90b822b147fd1105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:24.606683 2552 raft_consensus.cc:1261] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Refusing update from remote peer ec32d73bdb9d430c90b822b147fd1105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:24.606729 2631 consensus_queue.cc:1035] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:24.606884 2692 consensus_queue.cc:1035] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:24.616633 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:24.616729 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:24.617369 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735: Sending DeleteTablet for 3 replicas of tablet 7abd745b7128493380b2e6a7c9d80334
I20260620 14:24:24.617619 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abd745b7128493380b2e6a7c9d80334 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41940
I20260620 14:24:24.617624 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abd745b7128493380b2e6a7c9d80334 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35330
I20260620 14:24:24.617722 2698 tablet_replica.cc:330] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105: stopping tablet replica
I20260620 14:24:24.617727 2700 tablet_replica.cc:330] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de: stopping tablet replica
I20260620 14:24:24.617765 2698 raft_consensus.cc:2229] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:24.617779 2700 raft_consensus.cc:2229] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:24.617774 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abd745b7128493380b2e6a7c9d80334 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44622
I20260620 14:24:24.617820 2700 raft_consensus.cc:2258] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:24.617816 2698 raft_consensus.cc:2258] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:24.617908 2699 tablet_replica.cc:330] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c: stopping tablet replica
I20260620 14:24:24.617970 2699 raft_consensus.cc:2229] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:24.618008 2699 raft_consensus.cc:2258] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:24.618041 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:24:24.618113 2700 ts_tablet_manager.cc:1905] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:24:24.618194 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:24.618434 2699 ts_tablet_manager.cc:1905] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:24.619374 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab7aeeff5e94d93b366d93c59688cc0 (DEFAULT_TABLE table=test-table [id=f61533e56b5d462ea3226d1270b0442a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:24.619488 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab7aeeff5e94d93b366d93c59688cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:24.619781 2700 ts_tablet_manager.cc:1918] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:24.619839 2700 log.cc:1196] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-0/wal/wals/7abd745b7128493380b2e6a7c9d80334
I20260620 14:24:24.619975 2699 ts_tablet_manager.cc:1918] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:24.620045 2699 log.cc:1196] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/7abd745b7128493380b2e6a7c9d80334
I20260620 14:24:24.620060 2700 ts_tablet_manager.cc:1939] T 7abd745b7128493380b2e6a7c9d80334 P 3f8f1efc97f94c7a8777472e8f88a0de: Deleting consensus metadata
I20260620 14:24:24.620293 2699 ts_tablet_manager.cc:1939] T 7abd745b7128493380b2e6a7c9d80334 P b912c67b38b84ea9a35b32682003c36c: Deleting consensus metadata
I20260620 14:24:24.620373 2100 catalog_manager.cc:4841] TS 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503): tablet 7abd745b7128493380b2e6a7c9d80334 (table test-table [id=ef957346fc0f41b3b2852dc77b15628b]) successfully deleted
I20260620 14:24:24.620565 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab7aeeff5e94d93b366d93c59688cc0 (DEFAULT_TABLE table=test-table [id=f61533e56b5d462ea3226d1270b0442a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:24.620604 2100 catalog_manager.cc:4841] TS b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309): tablet 7abd745b7128493380b2e6a7c9d80334 (table test-table [id=ef957346fc0f41b3b2852dc77b15628b]) successfully deleted
I20260620 14:24:24.620678 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab7aeeff5e94d93b366d93c59688cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:24.620676 2709 tablet_bootstrap.cc:492] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de: Bootstrap starting.
I20260620 14:24:24.621057 2709 tablet_bootstrap.cc:654] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:24.621321 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab7aeeff5e94d93b366d93c59688cc0 (DEFAULT_TABLE table=test-table [id=f61533e56b5d462ea3226d1270b0442a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:24.621433 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab7aeeff5e94d93b366d93c59688cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:24.621665 2709 tablet_bootstrap.cc:492] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de: No bootstrap required, opened a new log
I20260620 14:24:24.621711 2709 ts_tablet_manager.cc:1397] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:24.621819 2711 tablet_bootstrap.cc:492] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105: Bootstrap starting.
I20260620 14:24:24.621845 2709 raft_consensus.cc:348] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.621905 2709 raft_consensus.cc:374] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:24.621924 2709 raft_consensus.cc:729] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8f1efc97f94c7a8777472e8f88a0de, State: Initialized, Role: FOLLOWER
I20260620 14:24:24.621965 2709 consensus_queue.cc:260] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.622059 2709 ts_tablet_manager.cc:1428] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:24.622169 2711 tablet_bootstrap.cc:654] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:24.622452 2698 ts_tablet_manager.cc:1905] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:24.622783 2710 tablet_bootstrap.cc:492] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c: Bootstrap starting.
I20260620 14:24:24.622797 2711 tablet_bootstrap.cc:492] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105: No bootstrap required, opened a new log
I20260620 14:24:24.622838 2711 ts_tablet_manager.cc:1397] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:24.622931 2711 raft_consensus.cc:348] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.622985 2711 raft_consensus.cc:374] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:24.623008 2711 raft_consensus.cc:729] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec32d73bdb9d430c90b822b147fd1105, State: Initialized, Role: FOLLOWER
I20260620 14:24:24.623056 2711 consensus_queue.cc:260] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.623111 2710 tablet_bootstrap.cc:654] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:24.623163 2711 ts_tablet_manager.cc:1428] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:24.623986 2698 ts_tablet_manager.cc:1918] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:24.624004 2710 tablet_bootstrap.cc:492] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c: No bootstrap required, opened a new log
I20260620 14:24:24.624048 2698 log.cc:1196] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/7abd745b7128493380b2e6a7c9d80334
I20260620 14:24:24.624063 2710 ts_tablet_manager.cc:1397] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:24.624186 2710 raft_consensus.cc:348] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.624266 2710 raft_consensus.cc:374] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:24.624305 2710 raft_consensus.cc:729] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b912c67b38b84ea9a35b32682003c36c, State: Initialized, Role: FOLLOWER
I20260620 14:24:24.624302 2698 ts_tablet_manager.cc:1939] T 7abd745b7128493380b2e6a7c9d80334 P ec32d73bdb9d430c90b822b147fd1105: Deleting consensus metadata
I20260620 14:24:24.624356 2710 consensus_queue.cc:260] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.624444 2710 ts_tablet_manager.cc:1428] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:24.624636 2100 catalog_manager.cc:4841] TS ec32d73bdb9d430c90b822b147fd1105 (127.0.98.2:46237): tablet 7abd745b7128493380b2e6a7c9d80334 (table test-table [id=ef957346fc0f41b3b2852dc77b15628b]) successfully deleted
I20260620 14:24:24.860277 2715 raft_consensus.cc:482] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:24.860430 2715 raft_consensus.cc:504] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.860795 2715 leader_election.cc:288] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec32d73bdb9d430c90b822b147fd1105 (127.0.98.2:46237), b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309)
I20260620 14:24:24.865180 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.861319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53617 (local address 127.0.98.3:43309)
0620 14:24:24.861439 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.861441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.861450 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:24:24.861507 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.861507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.861518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.861584 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:24.861743 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.862510 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.864123 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.864778 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.864886 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.864904 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.865058 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.865102 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260620 14:24:24.865180 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.861243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38337 (local address 127.0.98.2:46237)
0620 14:24:24.861296 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.861298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.861325 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:24:24.861365 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.861366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.861377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.861460 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:24.861661 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.862473 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.864124 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.864753 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.864840 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.864856 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.865061 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.865107 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:24:24.865212 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.861027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46237 (local address 127.0.98.1:38337)
0620 14:24:24.861159 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:24.861172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:24.861230 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:24.861474 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:24.861476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:24.861478 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:24.861559 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.861561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.862566 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.862567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.864003 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.864007 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.864888 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.864890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.864971 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.864977 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:24:24.865041 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:24:24.865102 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":12,"thread_start_us":79,"threads_started":1}
I20260620 14:24:24.865221 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.861222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.98.1:53617)
0620 14:24:24.861384 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:24.861392 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:24.861444 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:24.861602 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:24.861603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:24.861605 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:24.861663 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.861664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.862610 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.862611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.864005 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.864007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.864896 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.864897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.864971 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.864977 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:24:24.865041 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:24:24.865102 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":65,"thread_start_us":89,"threads_started":1}
I20260620 14:24:24.865507 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab7aeeff5e94d93b366d93c59688cc0" candidate_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b912c67b38b84ea9a35b32682003c36c" is_pre_election: true
I20260620 14:24:24.865610 2552 raft_consensus.cc:2454] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f8f1efc97f94c7a8777472e8f88a0de in term 0.
I20260620 14:24:24.865660 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab7aeeff5e94d93b366d93c59688cc0" candidate_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec32d73bdb9d430c90b822b147fd1105" is_pre_election: true
I20260620 14:24:24.865752 2412 raft_consensus.cc:2454] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f8f1efc97f94c7a8777472e8f88a0de in term 0.
I20260620 14:24:24.865801 2204 leader_election.cc:302] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f8f1efc97f94c7a8777472e8f88a0de, b912c67b38b84ea9a35b32682003c36c; no voters:
I20260620 14:24:24.865962 2715 raft_consensus.cc:2781] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:24.866006 2715 raft_consensus.cc:482] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:24.866029 2715 raft_consensus.cc:3037] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:24.866845 2715 raft_consensus.cc:504] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.866977 2715 leader_election.cc:288] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [CANDIDATE]: Term 1 election: Requested vote from peers ec32d73bdb9d430c90b822b147fd1105 (127.0.98.2:46237), b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309)
I20260620 14:24:24.867115 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab7aeeff5e94d93b366d93c59688cc0" candidate_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec32d73bdb9d430c90b822b147fd1105"
I20260620 14:24:24.867141 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab7aeeff5e94d93b366d93c59688cc0" candidate_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b912c67b38b84ea9a35b32682003c36c"
I20260620 14:24:24.867197 2412 raft_consensus.cc:3037] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:24.867202 2552 raft_consensus.cc:3037] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:24.867971 2412 raft_consensus.cc:2454] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f8f1efc97f94c7a8777472e8f88a0de in term 1.
I20260620 14:24:24.867971 2552 raft_consensus.cc:2454] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f8f1efc97f94c7a8777472e8f88a0de in term 1.
I20260620 14:24:24.868144 2204 leader_election.cc:302] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f8f1efc97f94c7a8777472e8f88a0de, b912c67b38b84ea9a35b32682003c36c; no voters:
I20260620 14:24:24.868233 2715 raft_consensus.cc:2781] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:24.868389 2715 raft_consensus.cc:686] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [term 1 LEADER]: Becoming Leader. State: Replica: 3f8f1efc97f94c7a8777472e8f88a0de, State: Running, Role: LEADER
I20260620 14:24:24.868505 2715 consensus_queue.cc:237] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260620 14:24:24.869249 2115 catalog_manager.cc:5495] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8f1efc97f94c7a8777472e8f88a0de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } health_report { overall_health: UNKNOWN } } }
I20260620 14:24:24.909873 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.907253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.0.98.1:34503)
0620 14:24:24.907442 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.907443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.907450 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:24:24.907486 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.907487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.907506 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.907576 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:24.907680 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.908190 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.908856 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.909350 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.909405 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.909535 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:24.909639 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:24.909694 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.909727 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.909747 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":122,"threads_started":1}
I20260620 14:24:24.914827 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.912482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42267 (local address 127.0.98.3:43309)
0620 14:24:24.912541 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.912542 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.912688 (+ 146us) server_negotiation.cc:407] Connection header received
0620 14:24:24.912732 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.912733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.912747 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.912795 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:24.912920 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.913523 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.914239 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.914437 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.914486 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.914640 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:24.914665 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:24.914701 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.914752 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.914769 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:24:24.914862 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.912397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.98.1:42267)
0620 14:24:24.912629 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:24.912639 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:24.912690 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:24.912809 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:24.912810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:24.912811 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:24:24.912869 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.912871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.913590 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.913590 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.914172 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.914174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.914516 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.914516 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.914562 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.914574 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:24:24.914715 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:24:24.914716 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:24:24.914751 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 14:24:24.914784 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":146,"threads_started":1}
I20260620 14:24:24.917026 2552 raft_consensus.cc:1261] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Refusing update from remote peer 3f8f1efc97f94c7a8777472e8f88a0de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:24.917025 2412 raft_consensus.cc:1261] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Refusing update from remote peer 3f8f1efc97f94c7a8777472e8f88a0de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:24.917281 2715 consensus_queue.cc:1035] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [LEADER]: Connected to new peer: Peer: permanent_uuid: "b912c67b38b84ea9a35b32682003c36c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:24.917409 2728 consensus_queue.cc:1035] T 8ab7aeeff5e94d93b366d93c59688cc0 P 3f8f1efc97f94c7a8777472e8f88a0de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec32d73bdb9d430c90b822b147fd1105" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:24.924865 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.922303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58771 (local address 127.0.98.1:34503)
0620 14:24:24.922429 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:24.922431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:24.922480 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:24:24.922546 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:24.922547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:24.922556 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:24.922611 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:24.922709 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.923175 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.924229 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.924440 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.924511 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.924633 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:24.924664 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:24.924727 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:24.924781 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:24:24.924800 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 14:24:24.924914 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:24.922268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34503 (local address 127.0.98.3:58771)
0620 14:24:24.922426 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:24.922438 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:24.922496 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:24.922614 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:24.922615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:24.922616 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:24:24.922662 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.922664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.923253 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.923254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.924154 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:24.924156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:24.924510 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:24.924510 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:24.924566 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:24.924571 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:24:24.924729 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:24:24.924731 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:24:24.924778 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:24:24.924830 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":114,"threads_started":1}
W20260620 14:24:24.925213 2681 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:24:26.447875 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:26.447985 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:26.448738 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf3b0d2e20924e8eb2b2a749fd09a735: Sending DeleteTablet for 3 replicas of tablet 8ab7aeeff5e94d93b366d93c59688cc0
I20260620 14:24:26.449051 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab7aeeff5e94d93b366d93c59688cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44622
I20260620 14:24:26.449057 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab7aeeff5e94d93b366d93c59688cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35330
I20260620 14:24:26.449101 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab7aeeff5e94d93b366d93c59688cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41940
I20260620 14:24:26.449218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 2187
I20260620 14:24:26.450058 2763 tablet_replica.cc:330] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c: stopping tablet replica
I20260620 14:24:26.450140 2763 raft_consensus.cc:2229] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:26.450191 2763 raft_consensus.cc:2258] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:26.450316 2764 tablet_replica.cc:330] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105: stopping tablet replica
I20260620 14:24:26.450410 2764 raft_consensus.cc:2229] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:26.450448 2764 raft_consensus.cc:2258] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:26.450651 2763 ts_tablet_manager.cc:1905] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:26.452117 2764 ts_tablet_manager.cc:1905] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:26.452184 2763 ts_tablet_manager.cc:1918] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:24:26.452251 2763 log.cc:1196] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/8ab7aeeff5e94d93b366d93c59688cc0
I20260620 14:24:26.452558 2763 ts_tablet_manager.cc:1939] T 8ab7aeeff5e94d93b366d93c59688cc0 P b912c67b38b84ea9a35b32682003c36c: Deleting consensus metadata
I20260620 14:24:26.452912 2100 catalog_manager.cc:4841] TS b912c67b38b84ea9a35b32682003c36c (127.0.98.3:43309): tablet 8ab7aeeff5e94d93b366d93c59688cc0 (table test-table [id=f61533e56b5d462ea3226d1270b0442a]) successfully deleted
I20260620 14:24:26.453469 2764 ts_tablet_manager.cc:1918] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:24:26.453550 2764 log.cc:1196] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/8ab7aeeff5e94d93b366d93c59688cc0
I20260620 14:24:26.453748 2764 ts_tablet_manager.cc:1939] T 8ab7aeeff5e94d93b366d93c59688cc0 P ec32d73bdb9d430c90b822b147fd1105: Deleting consensus metadata
I20260620 14:24:26.454071 2100 catalog_manager.cc:4841] TS ec32d73bdb9d430c90b822b147fd1105 (127.0.98.2:46237): tablet 8ab7aeeff5e94d93b366d93c59688cc0 (table test-table [id=f61533e56b5d462ea3226d1270b0442a]) successfully deleted
W20260620 14:24:26.457012 2100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34503 (error 108)
I20260620 14:24:26.457310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 2327
I20260620 14:24:26.458103 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:26.457472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34503 (local address 127.0.0.1:35358)
0620 14:24:26.457987 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:26.458060 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":68,"threads_started":1}
W20260620 14:24:26.458237 2100 catalog_manager.cc:4584] TS 3f8f1efc97f94c7a8777472e8f88a0de (127.0.98.1:34503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ab7aeeff5e94d93b366d93c59688cc0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34503: connect: Connection refused (error 111)
I20260620 14:24:26.464746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 2467
I20260620 14:24:26.470896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 2083
2026-06-20T14:24:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6224 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:26 dist-test-slave-hfc3 krb5kdc[2769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:26 dist-test-slave-hfc3 krb5kdc[2769](info): set up 2 sockets
Jun 20 14:24:26 dist-test-slave-hfc3 krb5kdc[2769](info): commencing operation
krb5kdc: starting...
W20260620 14:24:27.526266 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:27 dist-test-slave-hfc3 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:27.651383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46719
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:27.735675 2782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:27.735924 2782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:27.735961 2782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:27.737130 2782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:27.737193 2782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:27.737221 2782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:27.737252 2782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:27.737283 2782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:27.738844 2782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:27.739116 2782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:27.739334 2782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:27.744355 2787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:27.744355 2790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:27.744509 2782 server_base.cc:1034] running on GCE node
W20260620 14:24:27.744563 2788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:27.744933 2782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:27.745196 2782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:27.746407 2782 hybrid_clock.cc:648] HybridClock initialized: now 1781965467746383 us; error 37 us; skew 500 ppm
Jun 20 14:24:27 dist-test-slave-hfc3 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:27.748999 2782 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:27.749472 2782 webserver.cc:460] Webserver started at http://127.0.98.62:37289/ using document root <none> and password file <none>
I20260620 14:24:27.749701 2782 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:27.749766 2782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:27.749917 2782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:27.751052 2782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "51251a86ae134bf3b6dbd49abfcced85"
format_stamp: "Formatted at 2026-06-20 14:24:27 on dist-test-slave-hfc3"
server_key: "3d5eb7915da635662c191f8eabc7f622"
server_key_iv: "41d9e1c9412f7eb74f7a7beef2c60ae5"
server_key_version: "encryptionkey@0"
I20260620 14:24:27.751408 2782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "51251a86ae134bf3b6dbd49abfcced85"
format_stamp: "Formatted at 2026-06-20 14:24:27 on dist-test-slave-hfc3"
server_key: "3d5eb7915da635662c191f8eabc7f622"
server_key_iv: "41d9e1c9412f7eb74f7a7beef2c60ae5"
server_key_version: "encryptionkey@0"
I20260620 14:24:27.752729 2782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:24:27.753477 2797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:27.753762 2782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:27.753865 2782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "51251a86ae134bf3b6dbd49abfcced85"
format_stamp: "Formatted at 2026-06-20 14:24:27 on dist-test-slave-hfc3"
server_key: "3d5eb7915da635662c191f8eabc7f622"
server_key_iv: "41d9e1c9412f7eb74f7a7beef2c60ae5"
server_key_version: "encryptionkey@0"
I20260620 14:24:27.753942 2782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:27.765993 2782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:27.768004 2782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:27.768175 2782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:27.772234 2782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46719
I20260620 14:24:27.772265 2849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46719 every 8 connection(s)
I20260620 14:24:27.772785 2782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:27.773468 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:27.775861 2850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85: Bootstrap starting.
I20260620 14:24:27.776347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 2782
I20260620 14:24:27.776444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:27.776589 2850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:27.776659 392 external_mini_cluster.cc:1428] Setting key 17749dbb778c1f4c063335a481eddc08
I20260620 14:24:27.776923 2850 log.cc:824] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:27.777663 2850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85: No bootstrap required, opened a new log
I20260620 14:24:27.780252 2850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51251a86ae134bf3b6dbd49abfcced85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46719 } }
I20260620 14:24:27.780418 2850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:27.780447 2850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51251a86ae134bf3b6dbd49abfcced85, State: Initialized, Role: FOLLOWER
I20260620 14:24:27.780565 2850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51251a86ae134bf3b6dbd49abfcced85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46719 } }
I20260620 14:24:27.780658 2850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:27.780690 2850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:27.780743 2850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:27.781278 2850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51251a86ae134bf3b6dbd49abfcced85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46719 } }
I20260620 14:24:27.781387 2850 leader_election.cc:302] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51251a86ae134bf3b6dbd49abfcced85; no voters:
I20260620 14:24:27.781543 2850 leader_election.cc:288] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:27.781621 2855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:27.781786 2850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:27.781821 2855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [term 1 LEADER]: Becoming Leader. State: Replica: 51251a86ae134bf3b6dbd49abfcced85, State: Running, Role: LEADER
I20260620 14:24:27.782045 2855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51251a86ae134bf3b6dbd49abfcced85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46719 } }
I20260620 14:24:27.782519 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51251a86ae134bf3b6dbd49abfcced85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51251a86ae134bf3b6dbd49abfcced85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46719 } } }
I20260620 14:24:27.782603 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:27.783221 2857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51251a86ae134bf3b6dbd49abfcced85. Latest consensus state: current_term: 1 leader_uuid: "51251a86ae134bf3b6dbd49abfcced85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51251a86ae134bf3b6dbd49abfcced85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46719 } } }
I20260620 14:24:27.783308 2857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85 [sys.catalog]: This master's current role is: LEADER
W20260620 14:24:27.783845 2870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:27.783905 2870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:27.783984 2864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:27.784178 2864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:24:27 dist-test-slave-hfc3 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:27.785770 2864 catalog_manager.cc:1347] Generated new cluster ID: 8ee6fce84cb94644bfca17c7d8940b60
I20260620 14:24:27.785825 2864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:27.788822 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.777358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.0.98.62:46719)
0620 14:24:27.777533 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:27.777542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:27.777574 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:24:27.778135 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:27.778144 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:27.778208 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:27.778405 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:27.779096 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.780538 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.781407 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.782066 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.782180 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.785113 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:27.785117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:27.785117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:27.785129 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:27.787229 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:27.787604 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:27.787607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:27.787607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:27.787715 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:27.787914 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:27.787915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:27.787916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:27.788229 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:27.788395 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:27.788473 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:24:27.788611 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260620 14:24:27.805399 2864 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:27.805949 2864 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:27.815670 2864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51251a86ae134bf3b6dbd49abfcced85: Generated new TSK 0
I20260620 14:24:27.815893 2864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:27.840922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46719
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:27.923400 2876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:27.923609 2876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:27.923632 2876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:27.924826 2876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:27.924870 2876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:27.924897 2876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:27.926426 2876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:27.926693 2876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:27.926925 2876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:27.932036 2884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:27.932037 2882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:27.932107 2881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:27.932363 2876 server_base.cc:1034] running on GCE node
I20260620 14:24:27.932567 2876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:27.932767 2876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:27.933939 2876 hybrid_clock.cc:648] HybridClock initialized: now 1781965467933923 us; error 29 us; skew 500 ppm
Jun 20 14:24:27 dist-test-slave-hfc3 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:27.936609 2876 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:27.937062 2876 webserver.cc:460] Webserver started at http://127.0.98.1:38105/ using document root <none> and password file <none>
I20260620 14:24:27.937247 2876 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:27.937285 2876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:27.937374 2876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:27.938428 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "61d9ab38fb024f198edb7906027d15ff"
format_stamp: "Formatted at 2026-06-20 14:24:27 on dist-test-slave-hfc3"
server_key: "6d2c411c4d7f09c5db15d619db7f2a46"
server_key_iv: "1e621de24b37f34400570292d1572e6c"
server_key_version: "encryptionkey@0"
I20260620 14:24:27.938755 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "61d9ab38fb024f198edb7906027d15ff"
format_stamp: "Formatted at 2026-06-20 14:24:27 on dist-test-slave-hfc3"
server_key: "6d2c411c4d7f09c5db15d619db7f2a46"
server_key_iv: "1e621de24b37f34400570292d1572e6c"
server_key_version: "encryptionkey@0"
I20260620 14:24:27.940189 2876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:27.940920 2891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:27.941075 2876 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:27.941167 2876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "61d9ab38fb024f198edb7906027d15ff"
format_stamp: "Formatted at 2026-06-20 14:24:27 on dist-test-slave-hfc3"
server_key: "6d2c411c4d7f09c5db15d619db7f2a46"
server_key_iv: "1e621de24b37f34400570292d1572e6c"
server_key_version: "encryptionkey@0"
I20260620 14:24:27.941219 2876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:27.969976 2876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:27.972595 2876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:27.972733 2876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:27.972946 2876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:27.973225 2876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:27.973258 2876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:27.973280 2876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:27.973294 2876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:27.979101 2876 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43657
I20260620 14:24:27.979182 3004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43657 every 8 connection(s)
I20260620 14:24:27.979576 2876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:24:27 dist-test-slave-hfc3 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:27.986411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 2876
I20260620 14:24:27.986483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:27.986692 392 external_mini_cluster.cc:1428] Setting key 47066b36675523eff13ffc33f155006c
I20260620 14:24:27.989684 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.980613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48165 (local address 127.0.98.62:46719)
0620 14:24:27.980677 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:27.980679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:27.980852 (+ 173us) server_negotiation.cc:407] Connection header received
0620 14:24:27.981910 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:27.981911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:27.981921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:27.981970 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:27.982676 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.983273 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.984349 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.984551 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.984609 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.986851 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:27.986853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:27.986853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:27.986859 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:27.988319 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:27.988739 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:27.988740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:27.988740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:27.988802 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:27.989051 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:27.989052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:27.989052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:27.989168 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:27.989236 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:27.989508 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:24:27.989594 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:24:27.989907 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:27.980585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46719 (local address 127.0.98.1:48165)
0620 14:24:27.980747 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:27.980782 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:27.981799 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:27.982032 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:27.982035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:27.982311 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:27.982590 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.982595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.983356 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.983357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.984248 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:27.984250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:27.984623 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:27.984624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:27.984726 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:27.985219 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:27.985228 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:27.986706 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:27.988427 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:27.988434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:27.988436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:27.988664 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:27.988866 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:27.988867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:27.988867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:27.988980 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:27.989237 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:27.989245 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:27.989380 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 14:24:27.989476 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:24:27.989640 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260620 14:24:27.990362 3005 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46719
I20260620 14:24:27.990468 3005 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:27.990643 3005 heartbeater.cc:511] Master 127.0.98.62:46719 requested a full tablet report, sending...
I20260620 14:24:27.991199 2814 ts_manager.cc:194] Registered new tserver with Master: 61d9ab38fb024f198edb7906027d15ff (127.0.98.1:43657)
I20260620 14:24:27.992242 2814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48165
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:28.011016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46719
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:28.093815 3010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:28.094035 3010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:28.094059 3010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:28.095305 3010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:28.095351 3010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:28.095384 3010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:28.096894 3010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:28.097085 3010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:28.097277 3010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:28.102628 3015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:28.102638 3010 server_base.cc:1034] running on GCE node
W20260620 14:24:28.102782 3018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:28.102674 3016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:28.103120 3010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:28.103401 3010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:28.104607 3010 hybrid_clock.cc:648] HybridClock initialized: now 1781965468104575 us; error 46 us; skew 500 ppm
Jun 20 14:24:28 dist-test-slave-hfc3 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:28.107779 3010 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:28.108234 3010 webserver.cc:460] Webserver started at http://127.0.98.2:45137/ using document root <none> and password file <none>
I20260620 14:24:28.108459 3010 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:28.108507 3010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:28.108611 3010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:28.109741 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "8793926e60cc434ab852c5bf2afb84da"
format_stamp: "Formatted at 2026-06-20 14:24:28 on dist-test-slave-hfc3"
server_key: "b9ec090518308b4d454c4b45bade1762"
server_key_iv: "d16685d3f288060fd05952d39602025e"
server_key_version: "encryptionkey@0"
I20260620 14:24:28.110075 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8793926e60cc434ab852c5bf2afb84da"
format_stamp: "Formatted at 2026-06-20 14:24:28 on dist-test-slave-hfc3"
server_key: "b9ec090518308b4d454c4b45bade1762"
server_key_iv: "d16685d3f288060fd05952d39602025e"
server_key_version: "encryptionkey@0"
I20260620 14:24:28.111646 3010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:28.112481 3025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:28.112668 3010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:28.112742 3010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "8793926e60cc434ab852c5bf2afb84da"
format_stamp: "Formatted at 2026-06-20 14:24:28 on dist-test-slave-hfc3"
server_key: "b9ec090518308b4d454c4b45bade1762"
server_key_iv: "d16685d3f288060fd05952d39602025e"
server_key_version: "encryptionkey@0"
I20260620 14:24:28.112799 3010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:28.123782 3010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:28.125857 3010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:28.125968 3010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:28.126150 3010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:28.126456 3010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:28.126490 3010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:28.126513 3010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:28.126524 3010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:28.133378 3010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43209
I20260620 14:24:28.133419 3138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43209 every 8 connection(s)
I20260620 14:24:28.133792 3010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:28.135700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 3010
I20260620 14:24:28.135776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:28.135982 392 external_mini_cluster.cc:1428] Setting key 93c6232f321aa1676f66616f90f43d48
Jun 20 14:24:28 dist-test-slave-hfc3 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:28.145018 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:28.134633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52741 (local address 127.0.98.62:46719)
0620 14:24:28.134687 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:28.134689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:28.134953 (+ 264us) server_negotiation.cc:407] Connection header received
0620 14:24:28.136022 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:28.136023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:28.136045 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:28.136095 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:28.136890 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:28.137395 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:28.138742 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:28.138953 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:28.139022 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:28.141780 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:28.141782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:28.141782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:28.141791 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:28.143518 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:28.143989 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:28.143991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:28.143991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:28.144052 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:28.144292 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:28.144293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:28.144294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:28.144451 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:28.144530 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:28.144796 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 14:24:28.144882 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:24:28.145187 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:28.134629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46719 (local address 127.0.98.2:52741)
0620 14:24:28.134796 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:28.134858 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:28.135911 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:28.136163 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:28.136167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:28.136511 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:28.136804 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:28.136809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:28.137480 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:28.137481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:28.138655 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:28.138659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:28.139042 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:28.139043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:28.139268 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:28.139970 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:28.139978 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:28.141684 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:28.143608 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:28.143614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:28.143615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:28.143865 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:28.144129 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:28.144130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:28.144130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:28.144228 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:28.144521 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:28.144525 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:28.144655 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 14:24:28.144743 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:24:28.144865 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:24:28.145714 3139 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46719
I20260620 14:24:28.145819 3139 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:28.145979 3139 heartbeater.cc:511] Master 127.0.98.62:46719 requested a full tablet report, sending...
I20260620 14:24:28.146404 2814 ts_manager.cc:194] Registered new tserver with Master: 8793926e60cc434ab852c5bf2afb84da (127.0.98.2:43209)
I20260620 14:24:28.146816 2814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:28.162207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46719
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:28.243763 3144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:28.243963 3144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:28.243985 3144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:28.245142 3144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:28.245187 3144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:28.245213 3144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:28.246743 3144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:28.246923 3144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:28.247112 3144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:28.252190 3150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:28.252293 3152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:28.252225 3149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:28.252411 3144 server_base.cc:1034] running on GCE node
I20260620 14:24:28.252615 3144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:28.252868 3144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:28.254017 3144 hybrid_clock.cc:648] HybridClock initialized: now 1781965468253982 us; error 47 us; skew 500 ppm
Jun 20 14:24:28 dist-test-slave-hfc3 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:28.256681 3144 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:28.257194 3144 webserver.cc:460] Webserver started at http://127.0.98.3:44529/ using document root <none> and password file <none>
I20260620 14:24:28.257396 3144 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:28.257438 3144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:28.257572 3144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:28.258654 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "fb404ec18c7547c490142009151ca150"
format_stamp: "Formatted at 2026-06-20 14:24:28 on dist-test-slave-hfc3"
server_key: "5e30a59938f6531e6578c0a747df6720"
server_key_iv: "6d4de7e1531dcf668a89cb35e1748358"
server_key_version: "encryptionkey@0"
I20260620 14:24:28.258994 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fb404ec18c7547c490142009151ca150"
format_stamp: "Formatted at 2026-06-20 14:24:28 on dist-test-slave-hfc3"
server_key: "5e30a59938f6531e6578c0a747df6720"
server_key_iv: "6d4de7e1531dcf668a89cb35e1748358"
server_key_version: "encryptionkey@0"
I20260620 14:24:28.260277 3144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:24:28.260983 3159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:28.261216 3144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:24:28.261307 3144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "fb404ec18c7547c490142009151ca150"
format_stamp: "Formatted at 2026-06-20 14:24:28 on dist-test-slave-hfc3"
server_key: "5e30a59938f6531e6578c0a747df6720"
server_key_iv: "6d4de7e1531dcf668a89cb35e1748358"
server_key_version: "encryptionkey@0"
I20260620 14:24:28.261379 3144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:28.272109 3144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:28.273820 3144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:28.273969 3144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:28.274186 3144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:28.274526 3144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:28.274562 3144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:28.274600 3144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:28.274627 3144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:28.280473 3144 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46335
I20260620 14:24:28.280527 3272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46335 every 8 connection(s)
I20260620 14:24:28.280821 3144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:28.287026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 3144
I20260620 14:24:28.287113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:28.287302 392 external_mini_cluster.cc:1428] Setting key 741a8fb312dc79344f52ea8d6df54d0a
Jun 20 14:24:28 dist-test-slave-hfc3 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:28.290670 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:28.281580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33569 (local address 127.0.98.62:46719)
0620 14:24:28.281635 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:28.281637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:28.281847 (+ 210us) server_negotiation.cc:407] Connection header received
0620 14:24:28.283075 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:28.283076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:28.283090 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:28.283152 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:28.283903 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:28.284414 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:28.285442 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:28.285649 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:28.285731 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:28.288123 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:28.288124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:28.288125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:28.288131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:28.289471 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:28.289841 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:28.289842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:28.289842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:28.289894 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:28.290088 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:28.290089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:28.290089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:28.290187 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:28.290247 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:28.290546 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 14:24:28.290608 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:24:28.290833 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:28.281587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46719 (local address 127.0.98.3:33569)
0620 14:24:28.281730 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:28.281763 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:28.282944 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:28.283221 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:28.283225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:28.283543 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:28.283811 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:28.283815 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:28.284508 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:28.284509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:28.285335 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:28.285337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:28.285740 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:28.285741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:28.285868 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:28.286415 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:28.286423 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:28.288027 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:28.289547 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:28.289552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:28.289554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:28.289776 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:28.289946 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:28.289946 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:28.289947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:28.290039 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:28.290253 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:28.290258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:28.290431 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 14:24:28.290517 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:24:28.290636 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260620 14:24:28.291294 3273 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46719
I20260620 14:24:28.291406 3273 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:28.291567 3273 heartbeater.cc:511] Master 127.0.98.62:46719 requested a full tablet report, sending...
I20260620 14:24:28.291903 2814 ts_manager.cc:194] Registered new tserver with Master: fb404ec18c7547c490142009151ca150 (127.0.98.3:46335)
I20260620 14:24:28.292276 2814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33569
I20260620 14:24:28.299042 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:24:28.310289 2854 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:28.309382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.0.98.62:46719)
0620 14:24:28.309468 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:28.309470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:28.309535 (+ 65us) server_negotiation.cc:407] Connection header received
0620 14:24:28.309584 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:28.309585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:28.309594 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:28.309645 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:28.310213 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54942: BlockingRecv error: recv got EOF from 127.0.0.1:54942 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:24:28.311121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 2876
I20260620 14:24:28.315735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 3010
I20260620 14:24:28.320456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 3144
I20260620 14:24:28.324667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 2782
2026-06-20T14:24:28Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1852 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:28 dist-test-slave-hfc3 krb5kdc[3285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:28 dist-test-slave-hfc3 krb5kdc[3285](info): set up 2 sockets
Jun 20 14:24:28 dist-test-slave-hfc3 krb5kdc[3285](info): commencing operation
krb5kdc: starting...
W20260620 14:24:29.370312 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:29 dist-test-slave-hfc3 krb5kdc[3285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:29.488993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41745
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:29.569916 3298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:29.570173 3298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:29.570209 3298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:29.571424 3298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:29.571491 3298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:29.571512 3298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:29.571533 3298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:29.571560 3298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:29.573086 3298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:29.573372 3298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:29.573591 3298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:29.578516 3298 server_base.cc:1034] running on GCE node
W20260620 14:24:29.578476 3303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:29.578476 3304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:29.578483 3306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:29.578910 3298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:29.579217 3298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:29.580387 3298 hybrid_clock.cc:648] HybridClock initialized: now 1781965469580375 us; error 37 us; skew 500 ppm
Jun 20 14:24:29 dist-test-slave-hfc3 krb5kdc[3285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:29.582878 3298 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:29.583305 3298 webserver.cc:460] Webserver started at http://127.0.98.62:38429/ using document root <none> and password file <none>
I20260620 14:24:29.583542 3298 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:29.583606 3298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:29.583750 3298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:29.584780 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "574efaad68d54452882bcbc839fade84"
format_stamp: "Formatted at 2026-06-20 14:24:29 on dist-test-slave-hfc3"
server_key: "5c43429320ea78ee5c63629f3a1ddb19"
server_key_iv: "86e3ff1a42c6e42625ddcae18926309f"
server_key_version: "encryptionkey@0"
I20260620 14:24:29.585139 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "574efaad68d54452882bcbc839fade84"
format_stamp: "Formatted at 2026-06-20 14:24:29 on dist-test-slave-hfc3"
server_key: "5c43429320ea78ee5c63629f3a1ddb19"
server_key_iv: "86e3ff1a42c6e42625ddcae18926309f"
server_key_version: "encryptionkey@0"
I20260620 14:24:29.586516 3298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:24:29.587324 3313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:29.587529 3298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:29.587626 3298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "574efaad68d54452882bcbc839fade84"
format_stamp: "Formatted at 2026-06-20 14:24:29 on dist-test-slave-hfc3"
server_key: "5c43429320ea78ee5c63629f3a1ddb19"
server_key_iv: "86e3ff1a42c6e42625ddcae18926309f"
server_key_version: "encryptionkey@0"
I20260620 14:24:29.587704 3298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:29.604390 3298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:29.606590 3298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:29.606755 3298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:29.610834 3298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41745
I20260620 14:24:29.610960 3365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41745 every 8 connection(s)
I20260620 14:24:29.611393 3298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:29.612622 3366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:29.614001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 3298
I20260620 14:24:29.614114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:29.614313 392 external_mini_cluster.cc:1428] Setting key 766968b90ac052c4764948b51037f133
I20260620 14:24:29.617130 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84: Bootstrap starting.
I20260620 14:24:29.617924 3366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:29.618225 3366 log.cc:824] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:29.618952 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84: No bootstrap required, opened a new log
I20260620 14:24:29.621224 3366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574efaad68d54452882bcbc839fade84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41745 } }
I20260620 14:24:29.621387 3366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:29.621439 3366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574efaad68d54452882bcbc839fade84, State: Initialized, Role: FOLLOWER
I20260620 14:24:29.621569 3366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574efaad68d54452882bcbc839fade84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41745 } }
I20260620 14:24:29.621665 3366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:29.621706 3366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:29.621767 3366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:24:29 dist-test-slave-hfc3 krb5kdc[3285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:29.622398 3366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574efaad68d54452882bcbc839fade84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41745 } }
I20260620 14:24:29.622515 3366 leader_election.cc:302] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 574efaad68d54452882bcbc839fade84; no voters:
I20260620 14:24:29.622686 3366 leader_election.cc:288] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:29.622748 3371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:29.622897 3371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [term 1 LEADER]: Becoming Leader. State: Replica: 574efaad68d54452882bcbc839fade84, State: Running, Role: LEADER
I20260620 14:24:29.622985 3366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:29.623020 3371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574efaad68d54452882bcbc839fade84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41745 } }
I20260620 14:24:29.623327 3371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 574efaad68d54452882bcbc839fade84. Latest consensus state: current_term: 1 leader_uuid: "574efaad68d54452882bcbc839fade84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574efaad68d54452882bcbc839fade84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41745 } } }
I20260620 14:24:29.623427 3371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:29.623334 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "574efaad68d54452882bcbc839fade84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574efaad68d54452882bcbc839fade84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41745 } } }
I20260620 14:24:29.623565 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84 [sys.catalog]: This master's current role is: LEADER
W20260620 14:24:29.624815 3386 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:29.624869 3386 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:29.625113 3376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:29.625309 3376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:29.626794 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.614998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.0.98.62:41745)
0620 14:24:29.615257 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:29.615267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:29.615301 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:24:29.615932 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:29.615941 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:29.616024 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:29.616187 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:29.617092 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.618540 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.619373 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.619993 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.620116 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.622470 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:29.622475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:29.622476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:29.622488 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:29.624632 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:29.625126 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:29.625128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:29.625129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:29.625229 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:29.625467 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:29.625468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:29.625468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:29.625785 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:29.626327 (+ 542us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:29.626454 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:24:29.626583 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
I20260620 14:24:29.627019 3376 catalog_manager.cc:1347] Generated new cluster ID: 7358cce8af4345ab9c11d0d571247f86
I20260620 14:24:29.627071 3376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:29.627645 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:29.639556 3376 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:29.640304 3376 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:29.643942 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:29.640352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.0.98.62:41745)
0620 14:24:29.640394 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:29.640396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:29.640526 (+ 130us) server_negotiation.cc:407] Connection header received
0620 14:24:29.640612 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:29.640613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:29.640623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:29.640669 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:29.641039 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.641544 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.642308 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:29.642569 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:29.642642 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:29.643432 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:29.643433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:29.643433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:29.643639 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:29.643655 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:29.643661 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:29.643762 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:29.643819 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 14:24:29.643847 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:24:29.651944 3376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 574efaad68d54452882bcbc839fade84: Generated new TSK 0
I20260620 14:24:29.652151 3376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:29.654471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 3298
2026-06-20T14:24:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1327 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:29 dist-test-slave-hfc3 krb5kdc[3399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:29 dist-test-slave-hfc3 krb5kdc[3399](info): set up 2 sockets
Jun 20 14:24:29 dist-test-slave-hfc3 krb5kdc[3399](info): commencing operation
krb5kdc: starting...
W20260620 14:24:30.706880 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:30 dist-test-slave-hfc3 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:30.820503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39199
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:30.899129 3412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:30.899369 3412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:30.899406 3412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:30.900545 3412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:30.900610 3412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:30.900633 3412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:30.900655 3412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:30.900681 3412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:30.902112 3412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:30.902388 3412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:30.902619 3412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:30.907280 3417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:30.907265 3420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:30.907358 3418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:30.907600 3412 server_base.cc:1034] running on GCE node
I20260620 14:24:30.907835 3412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:30.908107 3412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:30.909261 3412 hybrid_clock.cc:648] HybridClock initialized: now 1781965470909253 us; error 34 us; skew 500 ppm
Jun 20 14:24:30 dist-test-slave-hfc3 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:30.911716 3412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:30.912175 3412 webserver.cc:460] Webserver started at http://127.0.98.62:46675/ using document root <none> and password file <none>
I20260620 14:24:30.912410 3412 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:30.912477 3412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:30.912622 3412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:30.913653 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "24b306ccc7794ed0a5e50ddab2c6ea7e"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-hfc3"
server_key: "02daba2b6c3927fa8787492f5b5c530a"
server_key_iv: "87e76665749f2b75b19ccfc808e0490f"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.914011 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "24b306ccc7794ed0a5e50ddab2c6ea7e"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-hfc3"
server_key: "02daba2b6c3927fa8787492f5b5c530a"
server_key_iv: "87e76665749f2b75b19ccfc808e0490f"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.915367 3412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:30.916138 3427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:30.916303 3412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:30.916410 3412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "24b306ccc7794ed0a5e50ddab2c6ea7e"
format_stamp: "Formatted at 2026-06-20 14:24:30 on dist-test-slave-hfc3"
server_key: "02daba2b6c3927fa8787492f5b5c530a"
server_key_iv: "87e76665749f2b75b19ccfc808e0490f"
server_key_version: "encryptionkey@0"
I20260620 14:24:30.916481 3412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:30.939035 3412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:30.940958 3412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:30.941118 3412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:30.945142 3412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39199
I20260620 14:24:30.945181 3479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39199 every 8 connection(s)
I20260620 14:24:30.945549 3412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:30.946573 3480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:30.948810 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e: Bootstrap starting.
I20260620 14:24:30.949393 3480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:30.949668 3480 log.cc:824] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:30.950322 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e: No bootstrap required, opened a new log
I20260620 14:24:30.951895 3480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b306ccc7794ed0a5e50ddab2c6ea7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39199 } }
I20260620 14:24:30.952054 3480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:30.952077 3480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b306ccc7794ed0a5e50ddab2c6ea7e, State: Initialized, Role: FOLLOWER
I20260620 14:24:30.952239 3480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b306ccc7794ed0a5e50ddab2c6ea7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39199 } }
I20260620 14:24:30.952316 3480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:30.952338 3480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:30.952399 3480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:30.953198 3480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b306ccc7794ed0a5e50ddab2c6ea7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39199 } }
I20260620 14:24:30.953285 3480 leader_election.cc:302] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24b306ccc7794ed0a5e50ddab2c6ea7e; no voters:
I20260620 14:24:30.953437 3480 leader_election.cc:288] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:30.953497 3483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:30.953701 3483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [term 1 LEADER]: Becoming Leader. State: Replica: 24b306ccc7794ed0a5e50ddab2c6ea7e, State: Running, Role: LEADER
I20260620 14:24:30.953691 3480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:30.953889 3483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b306ccc7794ed0a5e50ddab2c6ea7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39199 } }
I20260620 14:24:30.954269 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24b306ccc7794ed0a5e50ddab2c6ea7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b306ccc7794ed0a5e50ddab2c6ea7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39199 } } }
I20260620 14:24:30.954380 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:30.954568 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b306ccc7794ed0a5e50ddab2c6ea7e. Latest consensus state: current_term: 1 leader_uuid: "24b306ccc7794ed0a5e50ddab2c6ea7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b306ccc7794ed0a5e50ddab2c6ea7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39199 } } }
I20260620 14:24:30.954638 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:30.955037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 3412
I20260620 14:24:30.955153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:30.955364 392 external_mini_cluster.cc:1428] Setting key 28f0900146130dd0adad630571767920
I20260620 14:24:30.955607 3492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:30.956121 3492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:30.957688 3492 catalog_manager.cc:1347] Generated new cluster ID: e4769caacc6a493f9401ae14337698ea
I20260620 14:24:30.957741 3492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:24:30 dist-test-slave-hfc3 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:30.965881 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:30.956043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.0.98.62:39199)
0620 14:24:30.956243 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:30.956253 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:30.956284 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:24:30.956388 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:30.956393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:30.956444 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:30.956592 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:30.957384 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.958491 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.959260 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:30.959689 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:30.959803 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:30.962128 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:30.962132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:30.962132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:30.962144 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:30.964362 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:30.964726 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:30.964729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:30.964729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:30.964826 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:30.965027 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:30.965028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:30.965028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:30.965266 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:30.965460 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:30.965530 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:24:30.965677 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260620 14:24:31.010171 3492 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:31.010784 3492 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:31.022523 3492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24b306ccc7794ed0a5e50ddab2c6ea7e: Generated new TSK 0
I20260620 14:24:31.022742 3492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:31.053427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39199
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:31.134871 3506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:31.135121 3506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:31.135166 3506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:31.136693 3506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:31.136785 3506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:31.136837 3506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:31.138481 3506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:31.138751 3506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:31.138980 3506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:31.143702 3514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:31.143791 3512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:31.143963 3511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:31.144246 3506 server_base.cc:1034] running on GCE node
I20260620 14:24:31.144449 3506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:31.144706 3506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:31.145871 3506 hybrid_clock.cc:648] HybridClock initialized: now 1781965471145865 us; error 44 us; skew 500 ppm
Jun 20 14:24:31 dist-test-slave-hfc3 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:31.148403 3506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:31.148874 3506 webserver.cc:460] Webserver started at http://127.0.98.1:40899/ using document root <none> and password file <none>
I20260620 14:24:31.149101 3506 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:31.149165 3506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:31.149307 3506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:31.150318 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "b28ed2522b1441abbb2040291fa25f4d"
format_stamp: "Formatted at 2026-06-20 14:24:31 on dist-test-slave-hfc3"
server_key: "a660b605145dc85b7a584a03966fb4a5"
server_key_iv: "6473284c8f6fef3e7b3cf53ad40fb6bd"
server_key_version: "encryptionkey@0"
I20260620 14:24:31.150744 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b28ed2522b1441abbb2040291fa25f4d"
format_stamp: "Formatted at 2026-06-20 14:24:31 on dist-test-slave-hfc3"
server_key: "a660b605145dc85b7a584a03966fb4a5"
server_key_iv: "6473284c8f6fef3e7b3cf53ad40fb6bd"
server_key_version: "encryptionkey@0"
I20260620 14:24:31.152019 3506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:31.152706 3521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:31.152865 3506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:31.152962 3506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "b28ed2522b1441abbb2040291fa25f4d"
format_stamp: "Formatted at 2026-06-20 14:24:31 on dist-test-slave-hfc3"
server_key: "a660b605145dc85b7a584a03966fb4a5"
server_key_iv: "6473284c8f6fef3e7b3cf53ad40fb6bd"
server_key_version: "encryptionkey@0"
I20260620 14:24:31.153029 3506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:31.170660 3506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:31.173003 3506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:31.173166 3506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:31.173393 3506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:31.173708 3506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:31.173760 3506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:31.173802 3506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:31.173828 3506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:31.179714 3506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46825
I20260620 14:24:31.179759 3634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46825 every 8 connection(s)
I20260620 14:24:31.180073 3506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:24:31 dist-test-slave-hfc3 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:31.188766 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:31.180734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35479 (local address 127.0.98.62:39199)
0620 14:24:31.180778 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:31.180780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:31.180995 (+ 215us) server_negotiation.cc:407] Connection header received
0620 14:24:31.181491 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:31.181492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:31.181503 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:31.181562 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:31.182469 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.182974 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.183949 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.184135 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.184184 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:31.186162 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:31.186163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:31.186163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:31.186170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:31.187604 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:31.187991 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:31.187992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:31.187992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:31.188040 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:31.188242 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:31.188243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:31.188243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:31.188337 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:31.188385 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:31.188630 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:24:31.188701 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:24:31.188922 3637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:31.180711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39199 (local address 127.0.98.1:35479)
0620 14:24:31.180877 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:31.180919 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:31.181408 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:31.181616 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:31.181619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:31.182182 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:31.182391 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:31.182396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.183060 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.183061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:31.183878 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:31.183880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.184200 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.184201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:31.184302 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:31.184759 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:31.184767 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:31.186082 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:31.187677 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:31.187684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:31.187686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:31.187919 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:31.188089 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:31.188090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:31.188090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:31.188194 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:31.188390 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:31.188394 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:31.188514 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 14:24:31.188595 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:24:31.188708 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260620 14:24:31.189038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 3506
I20260620 14:24:31.189143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:31.189352 392 external_mini_cluster.cc:1428] Setting key 8c4a9c2f3e77e27150726029bc459e8f
I20260620 14:24:31.189519 3635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39199
I20260620 14:24:31.189617 3635 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:31.189795 3635 heartbeater.cc:511] Master 127.0.98.62:39199 requested a full tablet report, sending...
I20260620 14:24:31.190270 3444 ts_manager.cc:194] Registered new tserver with Master: b28ed2522b1441abbb2040291fa25f4d (127.0.98.1:46825)
I20260620 14:24:31.190932 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35479
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:31.213140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39199
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:31.293198 3640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:31.293412 3640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:31.293437 3640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:31.294571 3640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:31.294605 3640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:31.294631 3640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:31.295974 3640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:31.296165 3640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:31.296339 3640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:31.301277 3648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:31.301258 3645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:31.301236 3646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:31.301493 3640 server_base.cc:1034] running on GCE node
I20260620 14:24:31.301724 3640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:31.301965 3640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:31.303128 3640 hybrid_clock.cc:648] HybridClock initialized: now 1781965471303109 us; error 30 us; skew 500 ppm
Jun 20 14:24:31 dist-test-slave-hfc3 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:31.305711 3640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:31.306141 3640 webserver.cc:460] Webserver started at http://127.0.98.2:35537/ using document root <none> and password file <none>
I20260620 14:24:31.306432 3640 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:31.306497 3640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:31.306651 3640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:31.307803 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "0333c05a1b354960b11f900b1e67d84f"
format_stamp: "Formatted at 2026-06-20 14:24:31 on dist-test-slave-hfc3"
server_key: "4fe2acfb598b0614d6f02302336fc619"
server_key_iv: "6b2bf3c7eaad410f40fd3b34d3df5a51"
server_key_version: "encryptionkey@0"
I20260620 14:24:31.308157 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0333c05a1b354960b11f900b1e67d84f"
format_stamp: "Formatted at 2026-06-20 14:24:31 on dist-test-slave-hfc3"
server_key: "4fe2acfb598b0614d6f02302336fc619"
server_key_iv: "6b2bf3c7eaad410f40fd3b34d3df5a51"
server_key_version: "encryptionkey@0"
I20260620 14:24:31.309455 3640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:31.310186 3655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:31.310389 3640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:31.310492 3640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "0333c05a1b354960b11f900b1e67d84f"
format_stamp: "Formatted at 2026-06-20 14:24:31 on dist-test-slave-hfc3"
server_key: "4fe2acfb598b0614d6f02302336fc619"
server_key_iv: "6b2bf3c7eaad410f40fd3b34d3df5a51"
server_key_version: "encryptionkey@0"
I20260620 14:24:31.310602 3640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:31.332371 3640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:31.334270 3640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:31.334451 3640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:31.334686 3640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:31.334993 3640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:31.335058 3640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:31.335100 3640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:31.335129 3640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:31.340907 3640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45217
I20260620 14:24:31.340999 3768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45217 every 8 connection(s)
I20260620 14:24:31.341293 3640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:24:31 dist-test-slave-hfc3 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:31.348142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 3640
I20260620 14:24:31.348259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:31.348464 392 external_mini_cluster.cc:1428] Setting key 65c886d173a12c3efcda09281945ec33
I20260620 14:24:31.350467 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:31.342035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38191 (local address 127.0.98.62:39199)
0620 14:24:31.342080 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:31.342081 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:31.342285 (+ 204us) server_negotiation.cc:407] Connection header received
0620 14:24:31.342811 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:31.342812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:31.342822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:31.342870 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:31.343752 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.344254 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.345197 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.345386 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.345437 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:31.347504 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:31.347505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:31.347505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:31.347511 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:31.348989 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:31.349421 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:31.349423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:31.349423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:31.349488 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:31.349718 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:31.349719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:31.349719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:31.349873 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:31.349964 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:31.350202 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 14:24:31.350302 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:24:31.350530 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:31.342013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39199 (local address 127.0.98.2:38191)
0620 14:24:31.342171 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:31.342205 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:31.342724 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:31.342930 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:31.342934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:31.343488 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:31.343679 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:31.343684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.344329 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.344329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:31.345110 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:31.345111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.345452 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.345453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:31.345553 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:31.346063 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:31.346072 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:31.347420 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:31.349121 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:31.349127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:31.349129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:31.349348 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:31.349561 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:31.349562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:31.349562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:31.349656 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:31.349955 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:31.349958 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:31.350082 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 14:24:31.350163 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:24:31.350279 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260620 14:24:31.350955 3769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39199
I20260620 14:24:31.351058 3769 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:31.351231 3769 heartbeater.cc:511] Master 127.0.98.62:39199 requested a full tablet report, sending...
I20260620 14:24:31.351583 3444 ts_manager.cc:194] Registered new tserver with Master: 0333c05a1b354960b11f900b1e67d84f (127.0.98.2:45217)
I20260620 14:24:31.351946 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38191
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:31.371388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39199
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:31.451380 3774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:31.451581 3774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:31.451603 3774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:31.452677 3774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:31.452710 3774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:31.452736 3774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:31.454207 3774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:31.454484 3774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:31.454720 3774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:31.459880 3779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:31.459920 3774 server_base.cc:1034] running on GCE node
W20260620 14:24:31.460058 3780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:31.459913 3782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:31.460358 3774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:31.460614 3774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:31.461774 3774 hybrid_clock.cc:648] HybridClock initialized: now 1781965471461761 us; error 42 us; skew 500 ppm
Jun 20 14:24:31 dist-test-slave-hfc3 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:31.464183 3774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:31.464608 3774 webserver.cc:460] Webserver started at http://127.0.98.3:39297/ using document root <none> and password file <none>
I20260620 14:24:31.464843 3774 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:31.464908 3774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:31.465055 3774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:31.466105 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "d901915fbe8744adbf94a1c50119fb73"
format_stamp: "Formatted at 2026-06-20 14:24:31 on dist-test-slave-hfc3"
server_key: "96012aed6ed98c937794ceae5d7cdfb6"
server_key_iv: "24230151e2bf10506c8257a30be23027"
server_key_version: "encryptionkey@0"
I20260620 14:24:31.466480 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d901915fbe8744adbf94a1c50119fb73"
format_stamp: "Formatted at 2026-06-20 14:24:31 on dist-test-slave-hfc3"
server_key: "96012aed6ed98c937794ceae5d7cdfb6"
server_key_iv: "24230151e2bf10506c8257a30be23027"
server_key_version: "encryptionkey@0"
I20260620 14:24:31.467783 3774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:31.468443 3789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:31.468595 3774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:31.468698 3774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "d901915fbe8744adbf94a1c50119fb73"
format_stamp: "Formatted at 2026-06-20 14:24:31 on dist-test-slave-hfc3"
server_key: "96012aed6ed98c937794ceae5d7cdfb6"
server_key_iv: "24230151e2bf10506c8257a30be23027"
server_key_version: "encryptionkey@0"
I20260620 14:24:31.468777 3774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:31.507038 3774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:31.509045 3774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:31.509212 3774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:31.509442 3774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:31.509750 3774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:31.509804 3774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:31.509847 3774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:31.509886 3774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:31.515851 3774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41461
I20260620 14:24:31.515892 3902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41461 every 8 connection(s)
I20260620 14:24:31.516208 3774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:31.516765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 3774
I20260620 14:24:31.516832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:31.517033 392 external_mini_cluster.cc:1428] Setting key bc2b00c744f3a6b95dbee4847756f59c
Jun 20 14:24:31 dist-test-slave-hfc3 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:31.525516 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:31.516913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35209 (local address 127.0.98.62:39199)
0620 14:24:31.516985 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:31.516987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:31.517222 (+ 235us) server_negotiation.cc:407] Connection header received
0620 14:24:31.517901 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:31.517902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:31.517910 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:31.517953 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:31.518927 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.519442 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.520461 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.520636 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.520690 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:31.522681 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:31.522682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:31.522683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:31.522689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:31.524362 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:31.524724 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:31.524725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:31.524725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:31.524776 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:31.525014 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:31.525015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:31.525016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:31.525122 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:31.525181 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:31.525396 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 14:24:31.525455 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:24:31.525727 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:31.516887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39199 (local address 127.0.98.3:35209)
0620 14:24:31.517078 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:31.517129 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:31.517796 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:31.518026 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:31.518029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:31.518641 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:31.518826 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:31.518830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.519508 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.519509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:31.520391 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:31.520394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.520704 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.520704 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:31.520796 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:31.521235 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:31.521243 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:31.522583 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:31.524432 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:31.524437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:31.524439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:31.524663 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:31.524829 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:31.524829 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:31.524830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:31.524924 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:31.525181 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:31.525185 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:31.525292 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 14:24:31.525372 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:24:31.525481 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260620 14:24:31.526245 3903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39199
I20260620 14:24:31.526386 3903 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:31.526562 3903 heartbeater.cc:511] Master 127.0.98.62:39199 requested a full tablet report, sending...
I20260620 14:24:31.526975 3444 ts_manager.cc:194] Registered new tserver with Master: d901915fbe8744adbf94a1c50119fb73 (127.0.98.3:41461)
I20260620 14:24:31.527321 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35209
I20260620 14:24:31.528474 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:24:31 dist-test-slave-hfc3 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965471, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 14:24:31 dist-test-slave-hfc3 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965471, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:31.556018 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:31.544134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.0.98.62:39199)
0620 14:24:31.544193 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:31.544195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:31.544318 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:24:31.544369 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:31.544370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:31.544379 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:31.544427 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:31.544951 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.545420 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.546103 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:31.546276 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:31.546323 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:31.553376 (+ 7053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:31.553377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:31.553378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:31.553386 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:31.554925 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:31.555322 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:31.555323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:31.555323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:31.555389 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:31.555548 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:31.555548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:31.555548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:31.555701 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:31.555777 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:31.555866 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:24:31.555928 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 14:24:31.556272 3444 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46632
I20260620 14:24:31.556708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 3506
I20260620 14:24:31.561332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 3640
I20260620 14:24:31.565729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 3774
I20260620 14:24:31.569942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 3412
2026-06-20T14:24:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1917 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:31 dist-test-slave-hfc3 krb5kdc[3915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:31 dist-test-slave-hfc3 krb5kdc[3915](info): set up 2 sockets
Jun 20 14:24:31 dist-test-slave-hfc3 krb5kdc[3915](info): commencing operation
krb5kdc: starting...
W20260620 14:24:32.660615 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 14:24:32.830596 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.170s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 14:24:33.151506 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.321s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 14:24:33.296280 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.145s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/test-user.keytab.
W20260620 14:24:33.557245 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.261s user 0.000s sys 0.006s
Jun 20 14:24:33 dist-test-slave-hfc3 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:33.602324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41497
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:33.687258 3928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:33.687533 3928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:33.687558 3928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:33.688680 3928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:33.688717 3928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:33.688731 3928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:33.688745 3928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:33.688756 3928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:33.690116 3928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:33.690308 3928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:33.690553 3928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:33.696285 3936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:33.696334 3933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:33.696311 3934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:33.696460 3928 server_base.cc:1034] running on GCE node
I20260620 14:24:33.696799 3928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:33.697104 3928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:33.698269 3928 hybrid_clock.cc:648] HybridClock initialized: now 1781965473698261 us; error 42 us; skew 500 ppm
Jun 20 14:24:33 dist-test-slave-hfc3 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:33.700837 3928 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:33.701319 3928 webserver.cc:460] Webserver started at http://127.0.98.62:42561/ using document root <none> and password file <none>
I20260620 14:24:33.701555 3928 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:33.701624 3928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:33.701776 3928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:33.702920 3928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "2429146bf73242f3afa93b4fe37a6350"
format_stamp: "Formatted at 2026-06-20 14:24:33 on dist-test-slave-hfc3"
server_key: "7049fc8e1bc7b036840cf6da30152806"
server_key_iv: "fb00ca8aeea098ebdb4140004f0356d3"
server_key_version: "encryptionkey@0"
I20260620 14:24:33.703280 3928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "2429146bf73242f3afa93b4fe37a6350"
format_stamp: "Formatted at 2026-06-20 14:24:33 on dist-test-slave-hfc3"
server_key: "7049fc8e1bc7b036840cf6da30152806"
server_key_iv: "fb00ca8aeea098ebdb4140004f0356d3"
server_key_version: "encryptionkey@0"
I20260620 14:24:33.704583 3928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:24:33.705318 3943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:33.705518 3928 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:33.705624 3928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "2429146bf73242f3afa93b4fe37a6350"
format_stamp: "Formatted at 2026-06-20 14:24:33 on dist-test-slave-hfc3"
server_key: "7049fc8e1bc7b036840cf6da30152806"
server_key_iv: "fb00ca8aeea098ebdb4140004f0356d3"
server_key_version: "encryptionkey@0"
I20260620 14:24:33.705700 3928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:33.720824 3928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:33.723297 3928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:33.723482 3928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:33.727404 3928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41497
I20260620 14:24:33.727459 3995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41497 every 8 connection(s)
I20260620 14:24:33.727823 3928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:33.728686 3996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:33.730939 3996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350: Bootstrap starting.
I20260620 14:24:33.731462 3996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:33.731700 3996 log.cc:824] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:33.732319 3996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350: No bootstrap required, opened a new log
I20260620 14:24:33.733848 3996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2429146bf73242f3afa93b4fe37a6350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } }
I20260620 14:24:33.733982 3996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:33.734005 3996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2429146bf73242f3afa93b4fe37a6350, State: Initialized, Role: FOLLOWER
I20260620 14:24:33.734139 3996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2429146bf73242f3afa93b4fe37a6350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } }
I20260620 14:24:33.734222 3996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:33.734246 3996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:33.734273 3996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:33.734766 3996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2429146bf73242f3afa93b4fe37a6350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } }
I20260620 14:24:33.734858 3996 leader_election.cc:302] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2429146bf73242f3afa93b4fe37a6350; no voters:
I20260620 14:24:33.734978 3996 leader_election.cc:288] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:33.735057 3999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:33.735240 3996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:33.735455 3999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [term 1 LEADER]: Becoming Leader. State: Replica: 2429146bf73242f3afa93b4fe37a6350, State: Running, Role: LEADER
I20260620 14:24:33.735558 3999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2429146bf73242f3afa93b4fe37a6350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } }
I20260620 14:24:33.735931 4000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2429146bf73242f3afa93b4fe37a6350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2429146bf73242f3afa93b4fe37a6350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } } }
I20260620 14:24:33.735944 4001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2429146bf73242f3afa93b4fe37a6350. Latest consensus state: current_term: 1 leader_uuid: "2429146bf73242f3afa93b4fe37a6350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2429146bf73242f3afa93b4fe37a6350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } } }
I20260620 14:24:33.736050 4000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:33.736058 4001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:33.736250 4008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:33.736743 4008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:33.737579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 3928
I20260620 14:24:33.737689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:33.737912 392 external_mini_cluster.cc:1428] Setting key 5a63d6a431ed9a1cae26dcf01a3f022c
I20260620 14:24:33.738612 4008 catalog_manager.cc:1347] Generated new cluster ID: c5e8289c0d164145a0222ce33b0f060f
I20260620 14:24:33.738665 4008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:24:33 dist-test-slave-hfc3 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:33.749429 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:33.738452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.0.98.62:41497)
0620 14:24:33.738585 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:33.738594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:33.738627 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:24:33.738818 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:33.738825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:33.738886 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:33.739015 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:33.739785 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:33.741177 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:33.742253 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:33.742919 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:33.743013 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:33.745814 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:33.745819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:33.745819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:33.745832 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:33.747827 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:33.748196 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:33.748199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:33.748200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:33.748309 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:33.748507 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:33.748508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:33.748509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:33.748816 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:33.748978 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:33.749044 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:24:33.749180 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260620 14:24:33.763500 4008 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:33.763999 4008 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:33.772593 4008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2429146bf73242f3afa93b4fe37a6350: Generated new TSK 0
I20260620 14:24:33.772799 4008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:33.806551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41497
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:33.887619 4022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:33.887821 4022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:33.887844 4022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:33.889281 4022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:33.889334 4022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:33.889362 4022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:33.891229 4022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:33.891479 4022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:33.891760 4022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:33.896663 4027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:33.896649 4028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:33.896649 4030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:33.896862 4022 server_base.cc:1034] running on GCE node
I20260620 14:24:33.897107 4022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:33.897306 4022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:33.898396 4022 hybrid_clock.cc:648] HybridClock initialized: now 1781965473898377 us; error 33 us; skew 500 ppm
Jun 20 14:24:33 dist-test-slave-hfc3 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:33.900799 4022 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:33.901237 4022 webserver.cc:460] Webserver started at http://127.0.98.1:40035/ using document root <none> and password file <none>
I20260620 14:24:33.901422 4022 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:33.901463 4022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:33.901549 4022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:33.902595 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "11be72b8e3e748239322c68cad8e4085"
format_stamp: "Formatted at 2026-06-20 14:24:33 on dist-test-slave-hfc3"
server_key: "f8f70782c05d2a5487f54cd9612ae462"
server_key_iv: "80dbe6b5ce124319f279512847180154"
server_key_version: "encryptionkey@0"
I20260620 14:24:33.902906 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "11be72b8e3e748239322c68cad8e4085"
format_stamp: "Formatted at 2026-06-20 14:24:33 on dist-test-slave-hfc3"
server_key: "f8f70782c05d2a5487f54cd9612ae462"
server_key_iv: "80dbe6b5ce124319f279512847180154"
server_key_version: "encryptionkey@0"
I20260620 14:24:33.904167 4022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:24:33.904821 4037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:33.905033 4022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:33.905122 4022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "11be72b8e3e748239322c68cad8e4085"
format_stamp: "Formatted at 2026-06-20 14:24:33 on dist-test-slave-hfc3"
server_key: "f8f70782c05d2a5487f54cd9612ae462"
server_key_iv: "80dbe6b5ce124319f279512847180154"
server_key_version: "encryptionkey@0"
I20260620 14:24:33.905195 4022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:33.922669 4022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:33.924979 4022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:33.925140 4022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:33.925366 4022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:33.925675 4022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:33.925709 4022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:33.925730 4022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:33.925774 4022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:33.931586 4022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41041
I20260620 14:24:33.931631 4150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41041 every 8 connection(s)
I20260620 14:24:33.931988 4022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:24:33 dist-test-slave-hfc3 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:33.941392 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:33.932769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53923 (local address 127.0.98.62:41497)
0620 14:24:33.932845 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:33.932847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:33.933061 (+ 214us) server_negotiation.cc:407] Connection header received
0620 14:24:33.934014 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:33.934015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:33.934028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:33.934098 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:33.934852 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:33.935435 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:33.936549 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:33.936787 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:33.936848 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:33.938874 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:33.938876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:33.938876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:33.938884 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:33.940324 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:33.940651 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:33.940652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:33.940652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:33.940701 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:33.940897 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:33.940897 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:33.940897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:33.940991 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:33.941040 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:33.941253 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 14:24:33.941327 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:24:33.941617 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:33.932728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41497 (local address 127.0.98.1:53923)
0620 14:24:33.932933 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:33.932974 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:33.933889 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:33.934176 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:33.934179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:33.934517 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:33.934758 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:33.934763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:33.935514 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:33.935515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:33.936480 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:33.936482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:33.936865 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:33.936866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:33.936994 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:33.937443 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:33.937450 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:33.938789 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:33.940391 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:33.940396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:33.940398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:33.940589 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:33.940755 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:33.940756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:33.940756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:33.940843 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:33.941043 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:33.941047 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:33.941150 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 14:24:33.941228 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:24:33.941344 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":131,"threads_started":1}
I20260620 14:24:33.942070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 4022
I20260620 14:24:33.942152 4151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41497
I20260620 14:24:33.942188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:33.942250 4151 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:33.942420 392 external_mini_cluster.cc:1428] Setting key d2dd2da8ea77007eaddf66f34b00ce48
I20260620 14:24:33.942479 4151 heartbeater.cc:511] Master 127.0.98.62:41497 requested a full tablet report, sending...
I20260620 14:24:33.943064 3959 ts_manager.cc:194] Registered new tserver with Master: 11be72b8e3e748239322c68cad8e4085 (127.0.98.1:41041)
I20260620 14:24:33.943706 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53923
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:33.968858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41497
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:34.052263 4156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:34.052469 4156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:34.052491 4156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:34.053733 4156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:34.053773 4156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:34.053800 4156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:34.055306 4156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:34.055537 4156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:34.055768 4156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:34.060904 4156 server_base.cc:1034] running on GCE node
W20260620 14:24:34.060802 4161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:34.060822 4162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:34.060856 4164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:34.061269 4156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:34.061518 4156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:34.062659 4156 hybrid_clock.cc:648] HybridClock initialized: now 1781965474062644 us; error 29 us; skew 500 ppm
Jun 20 14:24:34 dist-test-slave-hfc3 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:34.065059 4156 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:34.065539 4156 webserver.cc:460] Webserver started at http://127.0.98.2:35361/ using document root <none> and password file <none>
I20260620 14:24:34.065776 4156 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:34.065843 4156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:34.065995 4156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:34.067091 4156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "d9cde6555a7a4fe6aa43ecd505f579fb"
format_stamp: "Formatted at 2026-06-20 14:24:34 on dist-test-slave-hfc3"
server_key: "bf3060b1d1c4774e662f4c891a5fa24c"
server_key_iv: "3be5f0860d2d74c4dfc272585fce25a6"
server_key_version: "encryptionkey@0"
I20260620 14:24:34.067462 4156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d9cde6555a7a4fe6aa43ecd505f579fb"
format_stamp: "Formatted at 2026-06-20 14:24:34 on dist-test-slave-hfc3"
server_key: "bf3060b1d1c4774e662f4c891a5fa24c"
server_key_iv: "3be5f0860d2d74c4dfc272585fce25a6"
server_key_version: "encryptionkey@0"
I20260620 14:24:34.068818 4156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:24:34.069604 4171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:34.069796 4156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:34.069905 4156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "d9cde6555a7a4fe6aa43ecd505f579fb"
format_stamp: "Formatted at 2026-06-20 14:24:34 on dist-test-slave-hfc3"
server_key: "bf3060b1d1c4774e662f4c891a5fa24c"
server_key_iv: "3be5f0860d2d74c4dfc272585fce25a6"
server_key_version: "encryptionkey@0"
I20260620 14:24:34.069978 4156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:34.082490 4156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:34.084564 4156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:34.084748 4156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:34.085016 4156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:34.085357 4156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:34.085414 4156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:34.085460 4156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:34.085497 4156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:34.092326 4156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44653
I20260620 14:24:34.092374 4284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44653 every 8 connection(s)
I20260620 14:24:34.092725 4156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:34.094141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 4156
I20260620 14:24:34.094274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:34.094496 392 external_mini_cluster.cc:1428] Setting key 951a4a9bfbee5d644c0566a330758866
Jun 20 14:24:34 dist-test-slave-hfc3 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:24:34.102840 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:34.093471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42241 (local address 127.0.98.62:41497)
0620 14:24:34.093526 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:34.093529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:34.093761 (+ 232us) server_negotiation.cc:407] Connection header received
0620 14:24:34.094975 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:34.094976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:34.094989 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:34.095032 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:34.095751 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:34.096263 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:34.097320 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:34.097534 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:34.097620 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:34.099934 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:34.099936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:34.099936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:34.099944 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:34.101373 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:34.101828 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:34.101829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:34.101830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:34.101890 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:34.102150 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:34.102150 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:34.102151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:34.102285 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:34.102384 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:34.102654 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 14:24:34.102742 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:24:34.103053 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:34.093466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41497 (local address 127.0.98.2:42241)
0620 14:24:34.093638 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:34.093674 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:34.094871 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:34.095109 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:34.095113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:34.095466 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:34.095658 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:34.095663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:34.096375 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:34.096377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:34.097223 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:34.097226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:34.097629 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:34.097630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:34.097767 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:34.098423 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:34.098431 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:34.099838 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:34.101483 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:34.101490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:34.101492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:34.101734 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:34.101958 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:34.101958 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:34.101959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:34.102073 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:34.102376 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:34.102383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:34.102526 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 14:24:34.102634 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:24:34.102806 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260620 14:24:34.103521 4285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41497
I20260620 14:24:34.103634 4285 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:34.103816 4285 heartbeater.cc:511] Master 127.0.98.62:41497 requested a full tablet report, sending...
I20260620 14:24:34.104194 3959 ts_manager.cc:194] Registered new tserver with Master: d9cde6555a7a4fe6aa43ecd505f579fb (127.0.98.2:44653)
I20260620 14:24:34.104636 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:34.120730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41497
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:34.202513 4290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:34.202723 4290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:34.202755 4290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:34.203979 4290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:34.204025 4290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:34.204052 4290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:34.205627 4290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:34.205890 4290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:34.206133 4290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:34.211362 4295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:34.211361 4298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:34.211444 4296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:34.211645 4290 server_base.cc:1034] running on GCE node
I20260620 14:24:34.211870 4290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:34.212111 4290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:34.213270 4290 hybrid_clock.cc:648] HybridClock initialized: now 1781965474213258 us; error 43 us; skew 500 ppm
Jun 20 14:24:34 dist-test-slave-hfc3 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:34.215970 4290 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:34.216351 4290 webserver.cc:460] Webserver started at http://127.0.98.3:34887/ using document root <none> and password file <none>
I20260620 14:24:34.216552 4290 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:34.216591 4290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:34.216674 4290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:34.217650 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "6d5c93b8252741f58c1bb0836cbc41ae"
format_stamp: "Formatted at 2026-06-20 14:24:34 on dist-test-slave-hfc3"
server_key: "e323abe3e658687c7de32db81c41a8f1"
server_key_iv: "1e09daadb8f5f8aa4e79cf69d5ba9d97"
server_key_version: "encryptionkey@0"
I20260620 14:24:34.217934 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6d5c93b8252741f58c1bb0836cbc41ae"
format_stamp: "Formatted at 2026-06-20 14:24:34 on dist-test-slave-hfc3"
server_key: "e323abe3e658687c7de32db81c41a8f1"
server_key_iv: "1e09daadb8f5f8aa4e79cf69d5ba9d97"
server_key_version: "encryptionkey@0"
I20260620 14:24:34.219449 4290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:34.220103 4305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:34.220286 4290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:34.220379 4290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "6d5c93b8252741f58c1bb0836cbc41ae"
format_stamp: "Formatted at 2026-06-20 14:24:34 on dist-test-slave-hfc3"
server_key: "e323abe3e658687c7de32db81c41a8f1"
server_key_iv: "1e09daadb8f5f8aa4e79cf69d5ba9d97"
server_key_version: "encryptionkey@0"
I20260620 14:24:34.220453 4290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:34.263746 4290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:34.266240 4290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:34.266508 4290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:34.266776 4290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:34.267100 4290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:34.267134 4290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:34.267158 4290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:34.267201 4290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:34.273381 4290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44451
I20260620 14:24:34.273428 4418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44451 every 8 connection(s)
I20260620 14:24:34.273743 4290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:34.275888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 4290
I20260620 14:24:34.275990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:34.276172 392 external_mini_cluster.cc:1428] Setting key c90981c9cc72425657c90792366b82db
Jun 20 14:24:34 dist-test-slave-hfc3 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:34.283586 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:34.274527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57139 (local address 127.0.98.62:41497)
0620 14:24:34.274583 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:34.274584 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:34.274808 (+ 224us) server_negotiation.cc:407] Connection header received
0620 14:24:34.275762 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:34.275763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:34.275772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:34.275817 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:34.276684 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:34.277303 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:34.278260 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:34.278512 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:34.278563 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:34.280555 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:34.280557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:34.280557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:34.280582 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:34.282107 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:34.282602 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:34.282603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:34.282604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:34.282659 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:34.282934 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:34.282935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:34.282936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:34.283057 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:34.283133 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:34.283415 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 14:24:34.283498 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:24:34.283752 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:34.274531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41497 (local address 127.0.98.3:57139)
0620 14:24:34.274682 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:34.274719 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:34.275641 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:34.275908 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:34.275912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:34.276295 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:34.276595 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:34.276601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:34.277383 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:34.277384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:34.278189 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:34.278191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:34.278582 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:34.278583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:34.278686 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:34.279113 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:34.279120 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:34.280460 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:34.282217 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:34.282224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:34.282226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:34.282515 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:34.282733 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:34.282734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:34.282734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:34.282845 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:34.283124 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:34.283130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:34.283267 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 14:24:34.283370 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:24:34.283522 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260620 14:24:34.284238 4419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41497
I20260620 14:24:34.284328 4419 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:34.284468 4419 heartbeater.cc:511] Master 127.0.98.62:41497 requested a full tablet report, sending...
I20260620 14:24:34.284806 3959 ts_manager.cc:194] Registered new tserver with Master: 6d5c93b8252741f58c1bb0836cbc41ae (127.0.98.3:44451)
I20260620 14:24:34.285132 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57139
I20260620 14:24:34.287693 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:24:34 dist-test-slave-hfc3 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:24:34 dist-test-slave-hfc3 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:24:34.310894 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:34.303053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.0.98.1:41041)
0620 14:24:34.303247 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:34.303252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:34.303263 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:24:34.303288 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:34.303290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:34.303304 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:34.303370 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:34.303953 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:34.304659 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:34.305366 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:34.305672 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:34.305724 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:34.308072 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:34.308080 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:34.308080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:34.308087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:34.309818 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:34.310117 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:34.310120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:34.310120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:34.310183 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:34.310378 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:34.310378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:34.310379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:34.310525 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:34.310628 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:34.310696 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:24:34.310757 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":98,"threads_started":1}
Jun 20 14:24:34 dist-test-slave-hfc3 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:34.318847 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:34.312126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.0.98.62:41497)
0620 14:24:34.312199 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:34.312200 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:34.312210 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:34.312286 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:34.312287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:34.312297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:34.312353 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:34.312824 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:34.313315 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:34.313996 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:34.314184 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:34.314266 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:34.316444 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:34.316446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:34.316446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:34.316454 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:34.317621 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:34.317982 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:34.317984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:34.317984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:34.318042 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:34.318241 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:34.318242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:34.318243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:34.318458 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:34.318568 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:34.318673 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:24:34.318748 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 14:24:34.319119 3959 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58970
I20260620 14:24:34.319491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 4022
I20260620 14:24:34.324438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 4156
I20260620 14:24:34.329718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 4290
I20260620 14:24:34.336699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 3928
2026-06-20T14:24:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2767 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:34 dist-test-slave-hfc3 krb5kdc[4431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:34 dist-test-slave-hfc3 krb5kdc[4431](info): set up 2 sockets
Jun 20 14:24:34 dist-test-slave-hfc3 krb5kdc[4431](info): commencing operation
krb5kdc: starting...
W20260620 14:24:35.378252 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:35 dist-test-slave-hfc3 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:35.494696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41233
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:35.575376 4444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:35.575583 4444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:35.575608 4444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:35.576792 4444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:35.576833 4444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:35.576848 4444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:35.576915 4444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:35.576956 4444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:35.578495 4444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:35.578758 4444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:35.578994 4444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:35.584002 4450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:35.583998 4449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:35.584028 4452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:35.584196 4444 server_base.cc:1034] running on GCE node
I20260620 14:24:35.584470 4444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:35.584707 4444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:35.585842 4444 hybrid_clock.cc:648] HybridClock initialized: now 1781965475585831 us; error 29 us; skew 500 ppm
Jun 20 14:24:35 dist-test-slave-hfc3 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:35.588553 4444 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:35.588928 4444 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:35.589013 4444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:35.589184 4444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:35.590231 4444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "35d9437e8bc54ee5a4256cb8567bbcb7"
format_stamp: "Formatted at 2026-06-20 14:24:35 on dist-test-slave-hfc3"
server_key: "273da8591f9c1388185db6aac1ace344"
server_key_iv: "fa9b22dcd7f4fdb2930bd8748178a543"
server_key_version: "encryptionkey@0"
I20260620 14:24:35.590659 4444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "35d9437e8bc54ee5a4256cb8567bbcb7"
format_stamp: "Formatted at 2026-06-20 14:24:35 on dist-test-slave-hfc3"
server_key: "273da8591f9c1388185db6aac1ace344"
server_key_iv: "fa9b22dcd7f4fdb2930bd8748178a543"
server_key_version: "encryptionkey@0"
I20260620 14:24:35.591987 4444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:35.592751 4458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:35.592922 4444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:35.593004 4444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "35d9437e8bc54ee5a4256cb8567bbcb7"
format_stamp: "Formatted at 2026-06-20 14:24:35 on dist-test-slave-hfc3"
server_key: "273da8591f9c1388185db6aac1ace344"
server_key_iv: "fa9b22dcd7f4fdb2930bd8748178a543"
server_key_version: "encryptionkey@0"
I20260620 14:24:35.593060 4444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:35.610034 4444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:35.613037 4444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:35.613220 4444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:35.618584 4444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41233
I20260620 14:24:35.618609 4510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41233 every 8 connection(s)
I20260620 14:24:35.618938 4444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:35.619323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 4444
I20260620 14:24:35.619410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:35.619617 392 external_mini_cluster.cc:1428] Setting key 0d17827335b639a232779c80eb86c96e
I20260620 14:24:35.620131 4511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:35.623581 4511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7: Bootstrap starting.
I20260620 14:24:35.624267 4511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:35.624557 4511 log.cc:824] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:35.625602 4511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7: No bootstrap required, opened a new log
Jun 20 14:24:35 dist-test-slave-hfc3 krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:35.627750 4511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d9437e8bc54ee5a4256cb8567bbcb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41233 } }
I20260620 14:24:35.627990 4511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:35.628152 4511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d9437e8bc54ee5a4256cb8567bbcb7, State: Initialized, Role: FOLLOWER
I20260620 14:24:35.628377 4511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d9437e8bc54ee5a4256cb8567bbcb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41233 } }
I20260620 14:24:35.628495 4511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:35.628542 4511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:35.628651 4511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:35.629210 4511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d9437e8bc54ee5a4256cb8567bbcb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41233 } }
I20260620 14:24:35.629336 4511 leader_election.cc:302] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35d9437e8bc54ee5a4256cb8567bbcb7; no voters:
I20260620 14:24:35.629551 4511 leader_election.cc:288] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:35.629582 4516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:35.629719 4516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [term 1 LEADER]: Becoming Leader. State: Replica: 35d9437e8bc54ee5a4256cb8567bbcb7, State: Running, Role: LEADER
I20260620 14:24:35.629829 4511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:35.629894 4516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d9437e8bc54ee5a4256cb8567bbcb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41233 } }
I20260620 14:24:35.630455 4516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35d9437e8bc54ee5a4256cb8567bbcb7. Latest consensus state: current_term: 1 leader_uuid: "35d9437e8bc54ee5a4256cb8567bbcb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d9437e8bc54ee5a4256cb8567bbcb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41233 } } }
I20260620 14:24:35.630585 4516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:35.630769 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35d9437e8bc54ee5a4256cb8567bbcb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d9437e8bc54ee5a4256cb8567bbcb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41233 } } }
I20260620 14:24:35.630863 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:35.630944 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:35.620253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.0.98.62:41233)
0620 14:24:35.620426 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:35.620435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:35.620460 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:24:35.620903 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:35.620909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:35.620961 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:35.621074 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:35.621872 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:35.622856 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:35.623908 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:35.624322 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:35.624448 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:35.626980 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:35.626984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:35.626985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:35.627004 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:35.628908 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:35.629250 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:35.629253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:35.629253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:35.629376 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:35.629570 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:35.629571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:35.629571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:35.629875 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:35.630005 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:35.630534 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 14:24:35.630684 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
W20260620 14:24:35.631956 4532 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:35.632009 4532 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:35.632062 4525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:35.632207 4525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:35.633592 4525 catalog_manager.cc:1347] Generated new cluster ID: 391f17a6a3c040f99b04683a2aa6e178
I20260620 14:24:35.633667 4525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:35.660269 4525 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:35.660832 4525 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:35.664745 4525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7: Generated new TSK 0
I20260620 14:24:35.664958 4525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:35.696852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41233
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:35.779088 4537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:35.779295 4537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:35.779320 4537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:35.780548 4537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:35.780598 4537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:35.780651 4537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:35.782223 4537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:35.782507 4537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:35.782740 4537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:35.787595 4543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:35.787596 4545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:35.787595 4542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:35.787854 4537 server_base.cc:1034] running on GCE node
I20260620 14:24:35.788065 4537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:35.788259 4537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:35.789419 4537 hybrid_clock.cc:648] HybridClock initialized: now 1781965475789405 us; error 28 us; skew 500 ppm
Jun 20 14:24:35 dist-test-slave-hfc3 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:35.792083 4537 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:35.792461 4537 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:35.792510 4537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:35.792654 4537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:35.793685 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "b2aee422c37c4737a2bafc8c0f4ece33"
format_stamp: "Formatted at 2026-06-20 14:24:35 on dist-test-slave-hfc3"
server_key: "9d5df92115e021c558cb8cd1bfb81f48"
server_key_iv: "4063f7b102066adaef60f9fe5332ea0b"
server_key_version: "encryptionkey@0"
I20260620 14:24:35.794029 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b2aee422c37c4737a2bafc8c0f4ece33"
format_stamp: "Formatted at 2026-06-20 14:24:35 on dist-test-slave-hfc3"
server_key: "9d5df92115e021c558cb8cd1bfb81f48"
server_key_iv: "4063f7b102066adaef60f9fe5332ea0b"
server_key_version: "encryptionkey@0"
I20260620 14:24:35.795346 4537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:35.796087 4551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:35.796276 4537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:35.796342 4537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "b2aee422c37c4737a2bafc8c0f4ece33"
format_stamp: "Formatted at 2026-06-20 14:24:35 on dist-test-slave-hfc3"
server_key: "9d5df92115e021c558cb8cd1bfb81f48"
server_key_iv: "4063f7b102066adaef60f9fe5332ea0b"
server_key_version: "encryptionkey@0"
I20260620 14:24:35.796423 4537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:35.810804 4537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:35.813162 4537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:35.813308 4537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:35.813546 4537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:35.813853 4537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:35.813887 4537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:35.813910 4537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:35.813954 4537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:35.819866 4537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45735
I20260620 14:24:35.819903 4664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45735 every 8 connection(s)
I20260620 14:24:35.820218 4537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:35.821888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 4537
I20260620 14:24:35.822000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:35.822183 392 external_mini_cluster.cc:1428] Setting key b777d30b3fca0bef72e1a6fb95923562
Jun 20 14:24:35 dist-test-slave-hfc3 krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:24:35.830498 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:35.820914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32945 (local address 127.0.98.62:41233)
0620 14:24:35.820992 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:35.820994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:35.821176 (+ 182us) server_negotiation.cc:407] Connection header received
0620 14:24:35.821623 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:35.821624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:35.821634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:35.821684 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:35.823021 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:35.823511 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:35.824883 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:35.825108 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:35.825200 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:35.827590 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:35.827591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:35.827592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:35.827600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:35.829002 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:35.829426 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:35.829427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:35.829428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:35.829491 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:35.829753 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:35.829754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:35.829754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:35.829930 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:35.829985 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:35.830297 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:24:35.830401 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:24:35.830703 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:35.820873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41233 (local address 127.0.98.1:32945)
0620 14:24:35.821053 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:35.821089 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:35.821540 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:35.821779 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:35.821783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:35.822197 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:35.822929 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:35.822936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:35.823604 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:35.823605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:35.824791 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:35.824794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:35.825212 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:35.825213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:35.825335 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:35.825895 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:35.825904 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:35.827489 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:35.829102 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:35.829109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:35.829111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:35.829343 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:35.829571 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:35.829572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:35.829572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:35.829681 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:35.829988 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:35.829995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:35.830166 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 14:24:35.830275 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:24:35.830465 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260620 14:24:35.831235 4665 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41233
I20260620 14:24:35.831346 4665 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:35.831516 4665 heartbeater.cc:511] Master 127.0.98.62:41233 requested a full tablet report, sending...
I20260620 14:24:35.832088 4475 ts_manager.cc:194] Registered new tserver with Master: b2aee422c37c4737a2bafc8c0f4ece33 (127.0.98.1:45735)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:24:35.832760 4475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:35.846261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41233
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:35.926556 4670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:35.926798 4670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:35.926838 4670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:35.927983 4670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:35.928056 4670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:35.928100 4670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:35.929711 4670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:35.929975 4670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:35.930227 4670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:35.935045 4678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:35.935122 4676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:35.935300 4675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:35.935395 4670 server_base.cc:1034] running on GCE node
I20260620 14:24:35.935628 4670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:35.935873 4670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:35.937068 4670 hybrid_clock.cc:648] HybridClock initialized: now 1781965475937061 us; error 41 us; skew 500 ppm
Jun 20 14:24:35 dist-test-slave-hfc3 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:35.939776 4670 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:35.940107 4670 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:35.940155 4670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:35.940244 4670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:35.941298 4670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "1f73b3114eac4b448829f44a659f55f6"
format_stamp: "Formatted at 2026-06-20 14:24:35 on dist-test-slave-hfc3"
server_key: "2660aac24ea613735b6c1e2947c490e6"
server_key_iv: "aef14055df986114535e958556274b2b"
server_key_version: "encryptionkey@0"
I20260620 14:24:35.941587 4670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1f73b3114eac4b448829f44a659f55f6"
format_stamp: "Formatted at 2026-06-20 14:24:35 on dist-test-slave-hfc3"
server_key: "2660aac24ea613735b6c1e2947c490e6"
server_key_iv: "aef14055df986114535e958556274b2b"
server_key_version: "encryptionkey@0"
I20260620 14:24:35.942886 4670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:35.943636 4684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:35.943871 4670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:35.943956 4670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "1f73b3114eac4b448829f44a659f55f6"
format_stamp: "Formatted at 2026-06-20 14:24:35 on dist-test-slave-hfc3"
server_key: "2660aac24ea613735b6c1e2947c490e6"
server_key_iv: "aef14055df986114535e958556274b2b"
server_key_version: "encryptionkey@0"
I20260620 14:24:35.944057 4670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:35.958465 4670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:35.960455 4670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:35.960568 4670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:35.960816 4670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:35.961189 4670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:35.961225 4670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:35.961269 4670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:35.961297 4670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:35.967175 4670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46519
I20260620 14:24:35.967257 4797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46519 every 8 connection(s)
I20260620 14:24:35.967532 4670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:35.970927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 4670
I20260620 14:24:35.971027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:35.971254 392 external_mini_cluster.cc:1428] Setting key 0c4a80e8648c3959714634036deebacc
Jun 20 14:24:35 dist-test-slave-hfc3 krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:35.978113 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:35.968280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36437 (local address 127.0.98.62:41233)
0620 14:24:35.968356 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:35.968358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:35.968563 (+ 205us) server_negotiation.cc:407] Connection header received
0620 14:24:35.969108 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:35.969109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:35.969122 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:35.969181 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:35.970160 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:35.970758 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:35.972062 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:35.972256 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:35.972329 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:35.974885 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:35.974886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:35.974886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:35.974893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:35.976486 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:35.977024 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:35.977025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:35.977025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:35.977075 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:35.977421 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:35.977422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:35.977423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:35.977542 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:35.977607 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:35.977979 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 14:24:35.978048 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:24:35.978466 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:35.968235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41233 (local address 127.0.98.2:36437)
0620 14:24:35.968416 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:35.968464 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:35.968987 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:35.969258 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:35.969262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:35.969546 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:35.970044 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:35.970049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:35.970856 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:35.970857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:35.971958 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:35.971963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:35.972333 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:35.972334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:35.972476 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:35.973038 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:35.973047 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:35.974779 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:35.976569 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:35.976575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:35.976578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:35.976908 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:35.977145 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:35.977146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:35.977147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:35.977326 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:35.977623 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:35.977629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:35.977824 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 14:24:35.977971 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 14:24:35.978176 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20260620 14:24:35.978977 4798 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41233
I20260620 14:24:35.979077 4798 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:35.979238 4798 heartbeater.cc:511] Master 127.0.98.62:41233 requested a full tablet report, sending...
I20260620 14:24:35.979620 4475 ts_manager.cc:194] Registered new tserver with Master: 1f73b3114eac4b448829f44a659f55f6 (127.0.98.2:46519)
I20260620 14:24:35.979962 4475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36437
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:35.998021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41233
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:36.080245 4803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:36.080511 4803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:36.080554 4803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:36.081758 4803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:36.081825 4803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:36.081871 4803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:36.083510 4803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:36.083793 4803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:36.084028 4803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:36.089164 4809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:36.089227 4811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:36.089387 4803 server_base.cc:1034] running on GCE node
W20260620 14:24:36.089249 4808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:36.089785 4803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:36.090083 4803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:36.091255 4803 hybrid_clock.cc:648] HybridClock initialized: now 1781965476091250 us; error 38 us; skew 500 ppm
Jun 20 14:24:36 dist-test-slave-hfc3 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:36.093916 4803 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:36.094413 4803 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:36.094494 4803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:36.094635 4803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:36.095798 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "127bfbf4fc024a24b724259a0397ed7a"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-hfc3"
server_key: "3fd0fd436af2ed72ef682b9488d78086"
server_key_iv: "20267e109030ff377c4913f8d3c93a7b"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.096171 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "127bfbf4fc024a24b724259a0397ed7a"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-hfc3"
server_key: "3fd0fd436af2ed72ef682b9488d78086"
server_key_iv: "20267e109030ff377c4913f8d3c93a7b"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.097522 4803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:36.098255 4817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:36.098477 4803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:36.098582 4803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "127bfbf4fc024a24b724259a0397ed7a"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-hfc3"
server_key: "3fd0fd436af2ed72ef682b9488d78086"
server_key_iv: "20267e109030ff377c4913f8d3c93a7b"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.098662 4803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:36.136169 4803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:36.138743 4803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:36.138911 4803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:36.139168 4803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:36.139521 4803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:36.139578 4803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:36.139622 4803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:36.139654 4803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:36.145565 4803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41269
I20260620 14:24:36.145601 4930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41269 every 8 connection(s)
I20260620 14:24:36.145874 4803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:24:36 dist-test-slave-hfc3 krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:36.154131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 4803
I20260620 14:24:36.154242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:36.154471 392 external_mini_cluster.cc:1428] Setting key 15fad76940d8c758c54201bea2fdaaac
I20260620 14:24:36.155416 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.146631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48755 (local address 127.0.98.62:41233)
0620 14:24:36.146755 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.146758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.146950 (+ 192us) server_negotiation.cc:407] Connection header received
0620 14:24:36.147434 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.147435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.147447 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.147506 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:36.148576 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.149077 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.150088 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.150293 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.150399 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.152445 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:36.152447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:36.152448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:36.152456 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:36.153688 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:36.154193 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:36.154194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:36.154195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:36.154307 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:36.154658 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:36.154660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:36.154660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:36.154807 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:36.154876 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.155228 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.155320 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 14:24:36.155599 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.146600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41233 (local address 127.0.98.3:48755)
0620 14:24:36.146810 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:36.146854 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:36.147329 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:36.147580 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:36.147584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:36.147950 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:36.148467 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:36.148473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.149172 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.149174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:36.149986 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:36.149988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.150408 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.150410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:36.150536 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.151017 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:36.151025 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:36.152345 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:36.153787 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:36.153794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:36.153796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:36.154109 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:36.154444 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:36.154445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:36.154445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:36.154584 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:36.154880 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:36.154886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:36.155040 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 14:24:36.155178 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 14:24:36.155353 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":118,"threads_started":1}
I20260620 14:24:36.156064 4931 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41233
I20260620 14:24:36.156162 4931 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:36.156272 4931 heartbeater.cc:511] Master 127.0.98.62:41233 requested a full tablet report, sending...
I20260620 14:24:36.156606 4475 ts_manager.cc:194] Registered new tserver with Master: 127bfbf4fc024a24b724259a0397ed7a (127.0.98.3:41269)
I20260620 14:24:36.156935 4475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48755
I20260620 14:24:36.166244 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:24:36.174081 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.167542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.0.98.62:41233)
0620 14:24:36.167636 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.167638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.167719 (+ 81us) server_negotiation.cc:407] Connection header received
0620 14:24:36.167797 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.167798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.167816 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.167879 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:36.168594 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.169085 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.169832 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.170063 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.170143 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.171393 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:36.171395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:36.171395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:36.171402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:36.172832 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:36.173223 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:36.173224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:36.173225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:36.173283 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:36.173502 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:36.173503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:36.173503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:36.173661 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:36.173772 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.173881 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.173972 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:24:36.175928 4475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:36.177603 4475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:36.184140 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.180095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45735 (local address 127.0.0.1:37536)
0620 14:24:36.180251 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:36.180294 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:36.180361 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:36.180650 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:36.180654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:36.180656 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:36.180736 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:36.180739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.181762 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.181763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:36.182882 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:36.182884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.183812 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.183813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:36.183892 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.183906 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:24:36.184008 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:24:36.184065 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260620 14:24:36.184154 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.180214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.0.98.1:45735)
0620 14:24:36.180408 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.180413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.180423 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:36.180462 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.180464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.180480 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.180550 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:36.180830 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.181653 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.182978 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.183711 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.183809 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.183872 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.184026 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.184066 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20260620 14:24:36.184621 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.180306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46519 (local address 127.0.0.1:53706)
0620 14:24:36.180759 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:36.180769 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:36.180811 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:36.181046 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:36.181047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:36.181049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:36.181101 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:36.181103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.182376 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.182377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:36.183306 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:36.183308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.184367 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.184368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:36.184437 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.184443 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:24:36.184497 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:24:36.184541 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":8,"thread_start_us":69,"threads_started":1}
I20260620 14:24:36.184720 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.180070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41269 (local address 127.0.0.1:58372)
0620 14:24:36.180252 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:36.180293 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:36.180361 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:36.180892 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:36.180893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:36.180895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:36.180952 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:36.180954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.182365 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.182367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:36.183492 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:36.183494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.184534 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.184534 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:36.184588 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.184591 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:36.184619 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 14:24:36.184657 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":45,"thread_start_us":62,"threads_started":1}
I20260620 14:24:36.185030 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.180464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.0.98.3:41269)
0620 14:24:36.180646 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.180650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.180660 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:36.180769 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.180772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.180790 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.181040 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:36.181126 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.182228 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.183578 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.184427 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.184816 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.184878 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.184901 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.184944 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":115,"threads_started":1}
I20260620 14:24:36.184911 4599 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd5ffdb81e1441aa763cf2058cbb864 (DEFAULT_TABLE table=test-table [id=643d5156feba4311bcf110f443a291dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:36.185070 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.180582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.0.98.2:46519)
0620 14:24:36.180877 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.180882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.180891 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:24:36.180930 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.180933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.180948 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.181033 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:36.181171 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.182248 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.183395 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.184125 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.184705 (+ 580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.184790 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.184818 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.184857 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260620 14:24:36.185328 4599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd5ffdb81e1441aa763cf2058cbb864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:36.187053 4732 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd5ffdb81e1441aa763cf2058cbb864 (DEFAULT_TABLE table=test-table [id=643d5156feba4311bcf110f443a291dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:36.187417 4732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd5ffdb81e1441aa763cf2058cbb864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:36.187328 4865 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd5ffdb81e1441aa763cf2058cbb864 (DEFAULT_TABLE table=test-table [id=643d5156feba4311bcf110f443a291dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:36.187629 4865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd5ffdb81e1441aa763cf2058cbb864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:36.188352 4948 tablet_bootstrap.cc:492] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33: Bootstrap starting.
I20260620 14:24:36.188977 4948 tablet_bootstrap.cc:654] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:36.189498 4948 log.cc:824] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:36.190205 4948 tablet_bootstrap.cc:492] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33: No bootstrap required, opened a new log
I20260620 14:24:36.190285 4948 ts_tablet_manager.cc:1397] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:36.190824 4950 tablet_bootstrap.cc:492] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6: Bootstrap starting.
I20260620 14:24:36.191254 4951 tablet_bootstrap.cc:492] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a: Bootstrap starting.
I20260620 14:24:36.191655 4950 tablet_bootstrap.cc:654] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:36.191834 4951 tablet_bootstrap.cc:654] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:36.192252 4950 log.cc:824] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:36.192338 4951 log.cc:824] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:36.192422 4948 raft_consensus.cc:348] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45735 } } peers { permanent_uuid: "1f73b3114eac4b448829f44a659f55f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46519 } } peers { permanent_uuid: "127bfbf4fc024a24b724259a0397ed7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41269 } }
I20260620 14:24:36.192631 4948 raft_consensus.cc:374] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:36.192693 4948 raft_consensus.cc:729] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2aee422c37c4737a2bafc8c0f4ece33, State: Initialized, Role: FOLLOWER
I20260620 14:24:36.192816 4948 consensus_queue.cc:260] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45735 } } peers { permanent_uuid: "1f73b3114eac4b448829f44a659f55f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46519 } } peers { permanent_uuid: "127bfbf4fc024a24b724259a0397ed7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41269 } }
I20260620 14:24:36.193107 4950 tablet_bootstrap.cc:492] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6: No bootstrap required, opened a new log
I20260620 14:24:36.193107 4948 ts_tablet_manager.cc:1428] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:24:36.193135 4665 heartbeater.cc:503] Master 127.0.98.62:41233 was elected leader, sending a full tablet report...
I20260620 14:24:36.193183 4950 ts_tablet_manager.cc:1397] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:36.194741 4951 tablet_bootstrap.cc:492] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a: No bootstrap required, opened a new log
I20260620 14:24:36.194811 4951 ts_tablet_manager.cc:1397] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 14:24:36.195004 4950 raft_consensus.cc:348] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45735 } } peers { permanent_uuid: "1f73b3114eac4b448829f44a659f55f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46519 } } peers { permanent_uuid: "127bfbf4fc024a24b724259a0397ed7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41269 } }
I20260620 14:24:36.195144 4950 raft_consensus.cc:374] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:36.195197 4950 raft_consensus.cc:729] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f73b3114eac4b448829f44a659f55f6, State: Initialized, Role: FOLLOWER
I20260620 14:24:36.195326 4950 consensus_queue.cc:260] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45735 } } peers { permanent_uuid: "1f73b3114eac4b448829f44a659f55f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46519 } } peers { permanent_uuid: "127bfbf4fc024a24b724259a0397ed7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41269 } }
I20260620 14:24:36.195606 4950 ts_tablet_manager.cc:1428] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:24:36.195695 4798 heartbeater.cc:503] Master 127.0.98.62:41233 was elected leader, sending a full tablet report...
I20260620 14:24:36.196673 4951 raft_consensus.cc:348] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45735 } } peers { permanent_uuid: "1f73b3114eac4b448829f44a659f55f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46519 } } peers { permanent_uuid: "127bfbf4fc024a24b724259a0397ed7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41269 } }
I20260620 14:24:36.196812 4951 raft_consensus.cc:374] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:36.196863 4951 raft_consensus.cc:729] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127bfbf4fc024a24b724259a0397ed7a, State: Initialized, Role: FOLLOWER
I20260620 14:24:36.196990 4951 consensus_queue.cc:260] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45735 } } peers { permanent_uuid: "1f73b3114eac4b448829f44a659f55f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46519 } } peers { permanent_uuid: "127bfbf4fc024a24b724259a0397ed7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41269 } }
I20260620 14:24:36.197232 4951 ts_tablet_manager.cc:1428] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:36.197306 4931 heartbeater.cc:503] Master 127.0.98.62:41233 was elected leader, sending a full tablet report...
W20260620 14:24:36.218070 4799 tablet.cc:2369] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:24:36.320884 4666 tablet.cc:2369] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:24:36.396378 4932 tablet.cc:2369] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:36.416390 4954 raft_consensus.cc:482] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:36.416515 4954 raft_consensus.cc:504] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45735 } } peers { permanent_uuid: "1f73b3114eac4b448829f44a659f55f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46519 } } peers { permanent_uuid: "127bfbf4fc024a24b724259a0397ed7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41269 } }
I20260620 14:24:36.416844 4954 leader_election.cc:288] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f73b3114eac4b448829f44a659f55f6 (127.0.98.2:46519), 127bfbf4fc024a24b724259a0397ed7a (127.0.98.3:41269)
I20260620 14:24:36.420979 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.417183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59993 (local address 127.0.98.3:41269)
0620 14:24:36.417231 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.417233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.417272 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:24:36.417378 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.417379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.417391 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.417466 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:36.417667 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.418437 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.420023 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.420603 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.420684 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.420694 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.420869 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.420907 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:24:36.420979 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.417157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44937 (local address 127.0.98.2:46519)
0620 14:24:36.417221 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.417223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.417272 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:24:36.417378 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.417379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.417392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.417466 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:36.417667 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.418437 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.420023 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.420603 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.420684 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.420694 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.420868 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.420907 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:24:36.421030 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.417062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41269 (local address 127.0.98.1:59993)
0620 14:24:36.417203 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:36.417216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:36.417300 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:36.417485 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:36.417485 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:36.417487 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:36.417592 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:36.417594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.418522 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.418523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:36.419917 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:36.419920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.420701 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.420702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:36.420781 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.420788 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:24:36.420848 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:24:36.420902 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":77,"thread_start_us":69,"threads_started":1}
I20260620 14:24:36.421033 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.417062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46519 (local address 127.0.98.1:44937)
0620 14:24:36.417205 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:36.417216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:36.417300 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:36.417485 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:36.417485 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:36.417489 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:36.417579 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:36.417582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.418522 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.418523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:36.419917 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:36.419920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.420697 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.420698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:36.420781 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.420787 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:24:36.420848 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:24:36.420902 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":91,"thread_start_us":59,"threads_started":1}
I20260620 14:24:36.421437 4885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd5ffdb81e1441aa763cf2058cbb864" candidate_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "127bfbf4fc024a24b724259a0397ed7a" is_pre_election: true
I20260620 14:24:36.421607 4885 raft_consensus.cc:2454] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2aee422c37c4737a2bafc8c0f4ece33 in term 0.
I20260620 14:24:36.421648 4752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd5ffdb81e1441aa763cf2058cbb864" candidate_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f73b3114eac4b448829f44a659f55f6" is_pre_election: true
I20260620 14:24:36.421818 4752 raft_consensus.cc:2454] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2aee422c37c4737a2bafc8c0f4ece33 in term 0.
I20260620 14:24:36.421871 4554 leader_election.cc:302] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 127bfbf4fc024a24b724259a0397ed7a, b2aee422c37c4737a2bafc8c0f4ece33; no voters:
I20260620 14:24:36.422000 4954 raft_consensus.cc:2781] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:36.422068 4954 raft_consensus.cc:482] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:36.422093 4954 raft_consensus.cc:3037] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:36.422820 4954 raft_consensus.cc:504] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45735 } } peers { permanent_uuid: "1f73b3114eac4b448829f44a659f55f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46519 } } peers { permanent_uuid: "127bfbf4fc024a24b724259a0397ed7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41269 } }
I20260620 14:24:36.422961 4954 leader_election.cc:288] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [CANDIDATE]: Term 1 election: Requested vote from peers 1f73b3114eac4b448829f44a659f55f6 (127.0.98.2:46519), 127bfbf4fc024a24b724259a0397ed7a (127.0.98.3:41269)
I20260620 14:24:36.423120 4885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd5ffdb81e1441aa763cf2058cbb864" candidate_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "127bfbf4fc024a24b724259a0397ed7a"
I20260620 14:24:36.423113 4752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd5ffdb81e1441aa763cf2058cbb864" candidate_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f73b3114eac4b448829f44a659f55f6"
I20260620 14:24:36.423192 4752 raft_consensus.cc:3037] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:36.423204 4885 raft_consensus.cc:3037] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:36.424093 4885 raft_consensus.cc:2454] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2aee422c37c4737a2bafc8c0f4ece33 in term 1.
I20260620 14:24:36.424093 4752 raft_consensus.cc:2454] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2aee422c37c4737a2bafc8c0f4ece33 in term 1.
I20260620 14:24:36.424332 4553 leader_election.cc:302] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f73b3114eac4b448829f44a659f55f6, b2aee422c37c4737a2bafc8c0f4ece33; no voters:
I20260620 14:24:36.424456 4954 raft_consensus.cc:2781] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:36.424669 4954 raft_consensus.cc:686] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [term 1 LEADER]: Becoming Leader. State: Replica: b2aee422c37c4737a2bafc8c0f4ece33, State: Running, Role: LEADER
I20260620 14:24:36.424786 4954 consensus_queue.cc:237] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45735 } } peers { permanent_uuid: "1f73b3114eac4b448829f44a659f55f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46519 } } peers { permanent_uuid: "127bfbf4fc024a24b724259a0397ed7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41269 } }
I20260620 14:24:36.425745 4475 catalog_manager.cc:5495] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2aee422c37c4737a2bafc8c0f4ece33 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2aee422c37c4737a2bafc8c0f4ece33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f73b3114eac4b448829f44a659f55f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "127bfbf4fc024a24b724259a0397ed7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41269 } health_report { overall_health: UNKNOWN } } }
I20260620 14:24:36.469090 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.466688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.0.98.1:45735)
0620 14:24:36.466759 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.466761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.466951 (+ 190us) server_negotiation.cc:407] Connection header received
0620 14:24:36.466998 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.466999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.467021 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.467095 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:36.467230 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.467682 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.468436 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:36.468629 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:36.468679 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:36.468829 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:36.468927 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:36.468987 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.469016 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.469032 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:24:36.470914 4752 raft_consensus.cc:1261] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6 [term 1 FOLLOWER]: Refusing update from remote peer b2aee422c37c4737a2bafc8c0f4ece33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:36.470953 4885 raft_consensus.cc:1261] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a [term 1 FOLLOWER]: Refusing update from remote peer b2aee422c37c4737a2bafc8c0f4ece33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:36.471244 4954 consensus_queue.cc:1035] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f73b3114eac4b448829f44a659f55f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:36.471371 4959 consensus_queue.cc:1035] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "127bfbf4fc024a24b724259a0397ed7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:36.476487 4474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:36.476591 4474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:36.477361 4474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35d9437e8bc54ee5a4256cb8567bbcb7: Sending DeleteTablet for 3 replicas of tablet 3dd5ffdb81e1441aa763cf2058cbb864
I20260620 14:24:36.477723 4865 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd5ffdb81e1441aa763cf2058cbb864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58372
I20260620 14:24:36.477753 4599 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd5ffdb81e1441aa763cf2058cbb864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37536
I20260620 14:24:36.477936 4971 tablet_replica.cc:330] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a: stopping tablet replica
I20260620 14:24:36.477998 4732 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd5ffdb81e1441aa763cf2058cbb864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53706
I20260620 14:24:36.478031 4971 raft_consensus.cc:2229] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:36.478135 4971 raft_consensus.cc:2258] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:36.478376 4973 tablet_replica.cc:330] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6: stopping tablet replica
I20260620 14:24:36.478387 4972 tablet_replica.cc:330] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33: stopping tablet replica
I20260620 14:24:36.478474 4972 raft_consensus.cc:2229] T 3dd5ffdb81e1441aa763cf2058cbb864 P b2aee422c37c4737a2bafc8c0f4ece33 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:36.478466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 4537
I20260620 14:24:36.478569 4973 raft_consensus.cc:2229] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:36.478684 4973 raft_consensus.cc:2258] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:36.479996 4973 ts_tablet_manager.cc:1905] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:36.480994 4971 ts_tablet_manager.cc:1905] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:36.481472 4973 ts_tablet_manager.cc:1918] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:36.481559 4973 log.cc:1196] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/3dd5ffdb81e1441aa763cf2058cbb864
I20260620 14:24:36.481786 4973 ts_tablet_manager.cc:1939] T 3dd5ffdb81e1441aa763cf2058cbb864 P 1f73b3114eac4b448829f44a659f55f6: Deleting consensus metadata
I20260620 14:24:36.482161 4460 catalog_manager.cc:4841] TS 1f73b3114eac4b448829f44a659f55f6 (127.0.98.2:46519): tablet 3dd5ffdb81e1441aa763cf2058cbb864 (table test-table [id=643d5156feba4311bcf110f443a291dc]) successfully deleted
I20260620 14:24:36.482944 4971 ts_tablet_manager.cc:1918] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:36.483022 4971 log.cc:1196] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/3dd5ffdb81e1441aa763cf2058cbb864
I20260620 14:24:36.483280 4971 ts_tablet_manager.cc:1939] T 3dd5ffdb81e1441aa763cf2058cbb864 P 127bfbf4fc024a24b724259a0397ed7a: Deleting consensus metadata
I20260620 14:24:36.483666 4461 catalog_manager.cc:4841] TS 127bfbf4fc024a24b724259a0397ed7a (127.0.98.3:41269): tablet 3dd5ffdb81e1441aa763cf2058cbb864 (table test-table [id=643d5156feba4311bcf110f443a291dc]) successfully deleted
W20260620 14:24:36.484400 4460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45735 (error 108)
I20260620 14:24:36.484599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 4670
I20260620 14:24:36.485329 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.485213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45735 (local address 127.0.0.1:37550)
0620 14:24:36.485266 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:36.485304 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:24:36.485456 4460 catalog_manager.cc:4584] TS b2aee422c37c4737a2bafc8c0f4ece33 (127.0.98.1:45735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3dd5ffdb81e1441aa763cf2058cbb864: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45735: connect: Connection refused (error 111)
I20260620 14:24:36.489692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 4803
I20260620 14:24:36.493870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 4444
2026-06-20T14:24:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2157 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T14:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:36Z Disabled control of system clock
I20260620 14:24:36.522075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42841
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:24:36.601132 4978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:36.601406 4978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:36.601444 4978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:36.602651 4978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:36.602735 4978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:36.602757 4978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:36.602777 4978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:36.602806 4978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:36.604178 4978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:36.604477 4978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:36.604794 4978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:36.609615 4984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:36.609710 4986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:36.609633 4983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:36.609769 4978 server_base.cc:1034] running on GCE node
I20260620 14:24:36.610059 4978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:36.610314 4978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:36.611502 4978 hybrid_clock.cc:648] HybridClock initialized: now 1781965476611492 us; error 27 us; skew 500 ppm
I20260620 14:24:36.616165 4978 webserver.cc:460] Webserver started at http://127.0.98.62:33307/ using document root <none> and password file <none>
I20260620 14:24:36.616456 4978 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:36.616506 4978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:36.616667 4978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:36.617779 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "cda5982365c149fea9d9cd33168ff56c"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-hfc3"
server_key: "4c976ba2ee727c0b5254620de15f502c"
server_key_iv: "3beda97a54fcfe6cfd25be8edd3c7554"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.618144 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "cda5982365c149fea9d9cd33168ff56c"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-hfc3"
server_key: "4c976ba2ee727c0b5254620de15f502c"
server_key_iv: "3beda97a54fcfe6cfd25be8edd3c7554"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.619544 4978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:36.620370 4992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:36.620613 4978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:36.620718 4978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "cda5982365c149fea9d9cd33168ff56c"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-hfc3"
server_key: "4c976ba2ee727c0b5254620de15f502c"
server_key_iv: "3beda97a54fcfe6cfd25be8edd3c7554"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.620805 4978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:36.624295 4978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:36.624571 4978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:36.624717 4978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:36.628518 4978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42841
I20260620 14:24:36.628552 5044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42841 every 8 connection(s)
I20260620 14:24:36.628880 4978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:36.629518 5045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:36.632227 5045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c: Bootstrap starting.
I20260620 14:24:36.632741 5045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:36.632975 5045 log.cc:824] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:36.633589 5045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c: No bootstrap required, opened a new log
I20260620 14:24:36.635213 5045 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5982365c149fea9d9cd33168ff56c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42841 } }
I20260620 14:24:36.635349 5045 raft_consensus.cc:374] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:36.635371 5045 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda5982365c149fea9d9cd33168ff56c, State: Initialized, Role: FOLLOWER
I20260620 14:24:36.635473 5045 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5982365c149fea9d9cd33168ff56c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42841 } }
I20260620 14:24:36.635524 5045 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:36.635543 5045 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:36.635610 5045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:36.636124 5045 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5982365c149fea9d9cd33168ff56c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42841 } }
I20260620 14:24:36.636209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 4978
I20260620 14:24:36.636250 5045 leader_election.cc:302] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda5982365c149fea9d9cd33168ff56c; no voters:
I20260620 14:24:36.636358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:36.636443 5045 leader_election.cc:288] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:36.636492 5048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:36.636560 392 external_mini_cluster.cc:1428] Setting key 66bd4188c4585621787e4827cb757a06
I20260620 14:24:36.636693 5048 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [term 1 LEADER]: Becoming Leader. State: Replica: cda5982365c149fea9d9cd33168ff56c, State: Running, Role: LEADER
I20260620 14:24:36.636765 5045 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:36.636781 5048 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5982365c149fea9d9cd33168ff56c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42841 } }
I20260620 14:24:36.637141 5048 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda5982365c149fea9d9cd33168ff56c. Latest consensus state: current_term: 1 leader_uuid: "cda5982365c149fea9d9cd33168ff56c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5982365c149fea9d9cd33168ff56c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42841 } } }
I20260620 14:24:36.637226 5048 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:36.637576 5049 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cda5982365c149fea9d9cd33168ff56c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5982365c149fea9d9cd33168ff56c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42841 } } }
I20260620 14:24:36.637652 5049 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:36.638204 5056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:36.639284 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.637175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.0.98.62:42841)
0620 14:24:36.637645 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.637654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.637692 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:24:36.637809 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.637817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.637862 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.638283 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:36.638851 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:36.638856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:36.638856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:36.638860 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:36.638903 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:36.638917 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:36.638988 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.639043 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.639104 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":88,"threads_started":1}
I20260620 14:24:36.639302 5056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:36.640908 5056 catalog_manager.cc:1347] Generated new cluster ID: 9ef69086ea1d428cb144bb895682e693
I20260620 14:24:36.640962 5056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:36.652112 5056 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:36.652328 5056 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:36.660786 5056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c: Generated new TSK 0
I20260620 14:24:36.661013 5056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:36.664637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42841
--builtin_ntp_servers=127.0.98.20:46311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:24:36.762902 5069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:36.763154 5069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:36.763187 5069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:36.764428 5069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:36.764499 5069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:36.764545 5069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:36.765935 5069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:36.766211 5069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:36.766572 5069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:36.772713 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:36.772708 5075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:36.772830 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:36.772733 5069 server_base.cc:1034] running on GCE node
I20260620 14:24:36.773241 5069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:36.773502 5069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:36.774668 5069 hybrid_clock.cc:648] HybridClock initialized: now 1781965476774653 us; error 33 us; skew 500 ppm
I20260620 14:24:36.776046 5069 webserver.cc:460] Webserver started at http://127.0.98.1:33373/ using document root <none> and password file <none>
I20260620 14:24:36.776297 5069 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:36.776369 5069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:36.776520 5069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:36.777612 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "1030f92396334ff187711f3a1dcb5a8d"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-hfc3"
server_key: "1a4e10ee13324afa8dc27c52503f4d84"
server_key_iv: "c8bba243929c89e221911be60ce6ce9f"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.777972 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1030f92396334ff187711f3a1dcb5a8d"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-hfc3"
server_key: "1a4e10ee13324afa8dc27c52503f4d84"
server_key_iv: "c8bba243929c89e221911be60ce6ce9f"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.779403 5069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:36.780175 5083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:36.780362 5069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:36.780460 5069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "1030f92396334ff187711f3a1dcb5a8d"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-hfc3"
server_key: "1a4e10ee13324afa8dc27c52503f4d84"
server_key_iv: "c8bba243929c89e221911be60ce6ce9f"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.780526 5069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:36.784790 5069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:36.785013 5069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:36.785109 5069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:36.785279 5069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:36.785524 5069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:36.785553 5069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:36.785574 5069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:36.785588 5069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:36.791076 5069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38861
I20260620 14:24:36.791145 5196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38861 every 8 connection(s)
I20260620 14:24:36.791469 5069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:36.793383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 5069
I20260620 14:24:36.793500 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:36.793666 392 external_mini_cluster.cc:1428] Setting key 30643ac4391860d0a7e856787a1567ae
I20260620 14:24:36.794720 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.792224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41941 (local address 127.0.98.62:42841)
0620 14:24:36.792286 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.792289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.792524 (+ 235us) server_negotiation.cc:407] Connection header received
0620 14:24:36.792977 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.792978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.792984 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.793039 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:36.794423 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:36.794424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:36.794425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:36.794429 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:36.794445 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:36.794449 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:36.794508 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.794604 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.794620 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:24:36.794807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42841
--builtin_ntp_servers=127.0.98.20:46311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:24:36.794821 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.792221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42841 (local address 127.0.98.1:41941)
0620 14:24:36.792411 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:36.792443 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:36.792896 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:36.793090 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:36.793093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:36.793101 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:36.794235 (+ 1134us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:36.794235 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:36.794271 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:36.794307 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:36.794504 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:36.794519 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:36.794522 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:36.794593 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:24:36.794640 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260620 14:24:36.795341 5197 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42841
I20260620 14:24:36.795408 5197 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:36.795507 5197 heartbeater.cc:511] Master 127.0.98.62:42841 requested a full tablet report, sending...
I20260620 14:24:36.795959 5009 ts_manager.cc:194] Registered new tserver with Master: 1030f92396334ff187711f3a1dcb5a8d (127.0.98.1:38861)
W20260620 14:24:36.875594 5200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:36.875803 5200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:36.875826 5200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:36.876993 5200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:36.877050 5200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:36.877105 5200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:36.878482 5200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:36.878727 5200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:36.879034 5200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:36.884053 5208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:36.884203 5200 server_base.cc:1034] running on GCE node
W20260620 14:24:36.884063 5206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:36.884053 5205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:36.884684 5200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:36.884920 5200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:36.886117 5200 hybrid_clock.cc:648] HybridClock initialized: now 1781965476886068 us; error 70 us; skew 500 ppm
I20260620 14:24:36.887526 5200 webserver.cc:460] Webserver started at http://127.0.98.2:41365/ using document root <none> and password file <none>
I20260620 14:24:36.887735 5200 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:36.887777 5200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:36.887866 5200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:36.888875 5200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-hfc3"
server_key: "fecd0c742131694fd08be554aa9d21be"
server_key_iv: "c44eb9149f6ddb94df8108d5153baa4c"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.889184 5200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-hfc3"
server_key: "fecd0c742131694fd08be554aa9d21be"
server_key_iv: "c44eb9149f6ddb94df8108d5153baa4c"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.890457 5200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:36.891141 5214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:36.891301 5200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:36.891373 5200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6"
format_stamp: "Formatted at 2026-06-20 14:24:36 on dist-test-slave-hfc3"
server_key: "fecd0c742131694fd08be554aa9d21be"
server_key_iv: "c44eb9149f6ddb94df8108d5153baa4c"
server_key_version: "encryptionkey@0"
I20260620 14:24:36.891428 5200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:36.894737 5200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:36.894963 5200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:36.895057 5200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:36.895223 5200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:36.895458 5200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:36.895488 5200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:36.895509 5200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:36.895521 5200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:36.901415 5200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34573
I20260620 14:24:36.901468 5327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34573 every 8 connection(s)
I20260620 14:24:36.901785 5200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:36.904590 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.902555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48509 (local address 127.0.98.62:42841)
0620 14:24:36.902599 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:36.902601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:36.902828 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:24:36.903254 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:36.903255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:36.903261 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:36.903313 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:36.904382 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:36.904384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:36.904384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:36.904389 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:36.904405 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:36.904410 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:36.904474 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:36.904521 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 14:24:36.904538 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:24:36.904794 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:36.902546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42841 (local address 127.0.98.2:48509)
0620 14:24:36.902696 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:36.902728 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:36.903174 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:36.903365 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:36.903368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:36.903375 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:36.904224 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:36.904225 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:36.904254 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:36.904280 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:36.904473 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:36.904476 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:36.904478 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:24:36.904527 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:24:36.904571 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260620 14:24:36.905239 5328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42841
I20260620 14:24:36.905335 5328 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:36.905493 5328 heartbeater.cc:511] Master 127.0.98.62:42841 requested a full tablet report, sending...
I20260620 14:24:36.905850 5009 ts_manager.cc:194] Registered new tserver with Master: 871b9e8c2c6845b2bf090eb5d39d7bd6 (127.0.98.2:34573)
I20260620 14:24:36.910027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 5200
I20260620 14:24:36.910133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:36.910319 392 external_mini_cluster.cc:1428] Setting key d4e7265e0b1b4365faa1cf7e80b70b94
I20260620 14:24:36.911257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42841
--builtin_ntp_servers=127.0.98.20:46311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:24:36.990954 5331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:36.991143 5331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:36.991165 5331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:36.992326 5331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:36.992365 5331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:36.992391 5331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:36.993752 5331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:36.994006 5331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:36.994259 5331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:36.999142 5339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:36.999137 5337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:36.999186 5336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:36.999379 5331 server_base.cc:1034] running on GCE node
I20260620 14:24:36.999567 5331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:36.999825 5331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:37.000968 5331 hybrid_clock.cc:648] HybridClock initialized: now 1781965477000957 us; error 24 us; skew 500 ppm
I20260620 14:24:37.002143 5331 webserver.cc:460] Webserver started at http://127.0.98.3:43997/ using document root <none> and password file <none>
I20260620 14:24:37.002465 5331 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:37.002535 5331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:37.002672 5331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:37.003682 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "d7bd5aab8f2749338e077d83cbe2806d"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "bee958b6d1388813a033cb694fbfcff3"
server_key_iv: "324a5ae6758b7170c72324c0ed30809c"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.004047 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d7bd5aab8f2749338e077d83cbe2806d"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "bee958b6d1388813a033cb694fbfcff3"
server_key_iv: "324a5ae6758b7170c72324c0ed30809c"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.005394 5331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:37.006100 5345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:37.006312 5331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:37.006433 5331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "d7bd5aab8f2749338e077d83cbe2806d"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "bee958b6d1388813a033cb694fbfcff3"
server_key_iv: "324a5ae6758b7170c72324c0ed30809c"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.006505 5331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:37.009656 5331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:37.009925 5331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:37.010076 5331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:37.010294 5331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:37.010620 5331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:37.010653 5331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:37.010711 5331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:37.010739 5331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:37.016254 5331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39509
I20260620 14:24:37.016328 5458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39509 every 8 connection(s)
I20260620 14:24:37.016649 5331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:37.019621 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.017375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39323 (local address 127.0.98.62:42841)
0620 14:24:37.017422 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.017423 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.017672 (+ 249us) server_negotiation.cc:407] Connection header received
0620 14:24:37.018135 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.018136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.018140 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.018206 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:37.019407 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:37.019408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:37.019408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:37.019412 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:37.019428 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:37.019432 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:37.019490 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:37.019555 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 14:24:37.019567 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:24:37.019831 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.017341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42841 (local address 127.0.98.3:39323)
0620 14:24:37.017538 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:37.017578 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:37.018056 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:37.018263 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:37.018266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:37.018274 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:37.019219 (+ 945us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:37.019220 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:37.019248 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:37.019285 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:37.019499 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:37.019503 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:37.019505 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:24:37.019562 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:24:37.019608 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260620 14:24:37.020361 5459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42841
I20260620 14:24:37.020447 5459 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:37.020584 5459 heartbeater.cc:511] Master 127.0.98.62:42841 requested a full tablet report, sending...
I20260620 14:24:37.020916 5009 ts_manager.cc:194] Registered new tserver with Master: d7bd5aab8f2749338e077d83cbe2806d (127.0.98.3:39509)
I20260620 14:24:37.025712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 5331
I20260620 14:24:37.025820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:37.025995 392 external_mini_cluster.cc:1428] Setting key 94c3729cfb12a2398a19e1436595e5d9
I20260620 14:24:37.026882 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:24:37.029577 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.028359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.0.98.62:42841)
0620 14:24:37.028406 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.028408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.028553 (+ 145us) server_negotiation.cc:407] Connection header received
0620 14:24:37.028636 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.028637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.028643 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.028730 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:37.029377 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:37.029378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:37.029378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:37.029381 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:37.029393 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:37.029397 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:37.029441 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:37.029470 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 14:24:37.029482 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:24:37.031373 5009 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:37.032893 5009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:37.036702 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.035014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39509 (local address 127.0.0.1:50912)
0620 14:24:37.035181 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:37.035219 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:37.035287 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:37.035699 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:37.035700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:37.035703 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:37.036191 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:37.036192 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:37.036208 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:37.036233 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:37.036472 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:37.036477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:37.036480 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:37.036556 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:24:37.036570 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260620 14:24:37.036968 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.035093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.0.98.1:38861)
0620 14:24:37.035248 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.035252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.035439 (+ 187us) server_negotiation.cc:407] Connection header received
0620 14:24:37.035506 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.035508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.035518 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.035592 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:37.036430 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:37.036434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:37.036435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:37.036438 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:37.036476 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:37.036492 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:37.036580 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:37.036865 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:24:37.036883 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260620 14:24:37.037086 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.035386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.0.98.3:39509)
0620 14:24:37.035539 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.035543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.035554 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:24:37.035578 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.035580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.035589 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.035880 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:37.036305 (+ 425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:37.036312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:37.036312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:37.036316 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:37.036350 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:37.036374 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:37.036472 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:37.036931 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 14:24:37.036946 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260620 14:24:37.037153 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.035017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38861 (local address 127.0.0.1:51784)
0620 14:24:37.035364 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:37.035373 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:37.035443 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:37.035603 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:37.035609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:37.035615 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:37.036338 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:37.036339 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:37.036345 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:37.036354 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:37.036605 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:37.036606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:37.036607 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:24:37.036647 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 14:24:37.036659 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":99,"threads_started":1}
I20260620 14:24:37.037542 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.035641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.0.98.2:34573)
0620 14:24:37.035834 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.035838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.035943 (+ 105us) server_negotiation.cc:407] Connection header received
0620 14:24:37.036033 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.036037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.036047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.036116 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:37.036906 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:37.036916 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:37.036916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:37.036921 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:37.036951 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:37.037016 (+ 65us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:37.037081 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:37.037457 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 14:24:37.037473 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":74,"threads_started":1}
I20260620 14:24:37.037748 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.035239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34573 (local address 127.0.0.1:55608)
0620 14:24:37.035885 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:37.035896 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:37.035964 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:37.036382 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:37.036383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:37.036385 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:37.036803 (+ 418us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:37.036803 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:37.036810 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:37.036822 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:37.037106 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:37.037107 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:37.037107 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 14:24:37.037674 (+ 567us) client_negotiation.cc:240] Negotiation successful
0620 14:24:37.037688 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":82,"threads_started":1}
I20260620 14:24:37.037809 5393 tablet_service.cc:1467] Processing CreateTablet for tablet cd5946b063c44168b4ce94b1d4b2a10e (DEFAULT_TABLE table=test-table [id=47169549d84241aa92395276c2f15003]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:37.037923 5131 tablet_service.cc:1467] Processing CreateTablet for tablet cd5946b063c44168b4ce94b1d4b2a10e (DEFAULT_TABLE table=test-table [id=47169549d84241aa92395276c2f15003]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:37.038183 5393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd5946b063c44168b4ce94b1d4b2a10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:37.038209 5131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd5946b063c44168b4ce94b1d4b2a10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:37.038400 5262 tablet_service.cc:1467] Processing CreateTablet for tablet cd5946b063c44168b4ce94b1d4b2a10e (DEFAULT_TABLE table=test-table [id=47169549d84241aa92395276c2f15003]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:37.038654 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd5946b063c44168b4ce94b1d4b2a10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:37.040916 5473 tablet_bootstrap.cc:492] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6: Bootstrap starting.
I20260620 14:24:37.041456 5475 tablet_bootstrap.cc:492] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d: Bootstrap starting.
I20260620 14:24:37.041697 5473 tablet_bootstrap.cc:654] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:37.042045 5473 log.cc:824] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:37.042150 5475 tablet_bootstrap.cc:654] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:37.042444 5474 tablet_bootstrap.cc:492] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d: Bootstrap starting.
I20260620 14:24:37.042512 5475 log.cc:824] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:37.042927 5473 tablet_bootstrap.cc:492] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6: No bootstrap required, opened a new log
I20260620 14:24:37.043001 5473 ts_tablet_manager.cc:1397] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:37.043085 5474 tablet_bootstrap.cc:654] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:37.043292 5475 tablet_bootstrap.cc:492] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d: No bootstrap required, opened a new log
I20260620 14:24:37.043366 5475 ts_tablet_manager.cc:1397] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:37.043393 5474 log.cc:824] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:37.044273 5474 tablet_bootstrap.cc:492] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d: No bootstrap required, opened a new log
I20260620 14:24:37.044346 5474 ts_tablet_manager.cc:1397] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:37.045279 5473 raft_consensus.cc:348] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1030f92396334ff187711f3a1dcb5a8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38861 } } peers { permanent_uuid: "d7bd5aab8f2749338e077d83cbe2806d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } } peers { permanent_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34573 } }
I20260620 14:24:37.045433 5473 raft_consensus.cc:374] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:37.045476 5473 raft_consensus.cc:729] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 871b9e8c2c6845b2bf090eb5d39d7bd6, State: Initialized, Role: FOLLOWER
I20260620 14:24:37.045603 5473 consensus_queue.cc:260] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1030f92396334ff187711f3a1dcb5a8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38861 } } peers { permanent_uuid: "d7bd5aab8f2749338e077d83cbe2806d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } } peers { permanent_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34573 } }
I20260620 14:24:37.045871 5473 ts_tablet_manager.cc:1428] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:24:37.046011 5328 heartbeater.cc:503] Master 127.0.98.62:42841 was elected leader, sending a full tablet report...
I20260620 14:24:37.045940 5475 raft_consensus.cc:348] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1030f92396334ff187711f3a1dcb5a8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38861 } } peers { permanent_uuid: "d7bd5aab8f2749338e077d83cbe2806d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } } peers { permanent_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34573 } }
I20260620 14:24:37.046103 5475 raft_consensus.cc:374] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:37.046154 5475 raft_consensus.cc:729] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1030f92396334ff187711f3a1dcb5a8d, State: Initialized, Role: FOLLOWER
I20260620 14:24:37.046454 5475 consensus_queue.cc:260] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1030f92396334ff187711f3a1dcb5a8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38861 } } peers { permanent_uuid: "d7bd5aab8f2749338e077d83cbe2806d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } } peers { permanent_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34573 } }
I20260620 14:24:37.046737 5197 heartbeater.cc:503] Master 127.0.98.62:42841 was elected leader, sending a full tablet report...
I20260620 14:24:37.046707 5475 ts_tablet_manager.cc:1428] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 14:24:37.047405 5474 raft_consensus.cc:348] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1030f92396334ff187711f3a1dcb5a8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38861 } } peers { permanent_uuid: "d7bd5aab8f2749338e077d83cbe2806d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } } peers { permanent_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34573 } }
I20260620 14:24:37.047544 5474 raft_consensus.cc:374] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:37.047591 5474 raft_consensus.cc:729] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7bd5aab8f2749338e077d83cbe2806d, State: Initialized, Role: FOLLOWER
I20260620 14:24:37.047698 5474 consensus_queue.cc:260] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1030f92396334ff187711f3a1dcb5a8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38861 } } peers { permanent_uuid: "d7bd5aab8f2749338e077d83cbe2806d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } } peers { permanent_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34573 } }
I20260620 14:24:37.047941 5474 ts_tablet_manager.cc:1428] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:24:37.047998 5459 heartbeater.cc:503] Master 127.0.98.62:42841 was elected leader, sending a full tablet report...
I20260620 14:24:37.115244 5479 raft_consensus.cc:482] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:37.115371 5479 raft_consensus.cc:504] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1030f92396334ff187711f3a1dcb5a8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38861 } } peers { permanent_uuid: "d7bd5aab8f2749338e077d83cbe2806d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } } peers { permanent_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34573 } }
I20260620 14:24:37.115715 5479 leader_election.cc:288] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1030f92396334ff187711f3a1dcb5a8d (127.0.98.1:38861), d7bd5aab8f2749338e077d83cbe2806d (127.0.98.3:39509)
I20260620 14:24:37.117117 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.116049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49753 (local address 127.0.98.3:39509)
0620 14:24:37.116170 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.116172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.116188 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:24:37.116212 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.116213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.116217 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.116296 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:37.116875 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:37.116876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:37.116877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:37.116881 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:37.116897 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:37.116907 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:37.116970 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:37.117019 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:24:37.117033 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 14:24:37.117159 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.115895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39509 (local address 127.0.98.2:49753)
0620 14:24:37.115954 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:37.115967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:37.116012 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:37.116306 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:37.116307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:37.116309 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:37.116758 (+ 449us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:37.116758 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:37.116765 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:37.116777 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:37.116967 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:37.116967 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:37.116968 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:24:37.117019 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:24:37.117034 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:24:37.117364 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.115953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50737 (local address 127.0.98.1:38861)
0620 14:24:37.116005 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.116007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.116257 (+ 250us) server_negotiation.cc:407] Connection header received
0620 14:24:37.116319 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.116320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.116334 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.116394 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:37.117046 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:37.117047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:37.117048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:37.117066 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:37.117080 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:37.117083 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:37.117164 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:37.117256 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:24:37.117272 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:24:37.117538 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.115911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38861 (local address 127.0.98.2:50737)
0620 14:24:37.116181 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:37.116189 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:37.116251 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:37.116400 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:37.116401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:37.116403 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:37.116947 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:24:37.116948 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:37.116954 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:24:37.116965 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:37.117160 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:37.117161 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:37.117162 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:24:37.117454 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 14:24:37.117469 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":81,"thread_start_us":136,"threads_started":1}
I20260620 14:24:37.117624 5413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5946b063c44168b4ce94b1d4b2a10e" candidate_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bd5aab8f2749338e077d83cbe2806d" is_pre_election: true
I20260620 14:24:37.117787 5413 raft_consensus.cc:2454] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 871b9e8c2c6845b2bf090eb5d39d7bd6 in term 0.
I20260620 14:24:37.117909 5151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5946b063c44168b4ce94b1d4b2a10e" candidate_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1030f92396334ff187711f3a1dcb5a8d" is_pre_election: true
I20260620 14:24:37.118000 5215 leader_election.cc:302] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 871b9e8c2c6845b2bf090eb5d39d7bd6, d7bd5aab8f2749338e077d83cbe2806d; no voters:
I20260620 14:24:37.118057 5151 raft_consensus.cc:2454] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 871b9e8c2c6845b2bf090eb5d39d7bd6 in term 0.
I20260620 14:24:37.118150 5479 raft_consensus.cc:2781] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:37.118216 5479 raft_consensus.cc:482] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:37.118254 5479 raft_consensus.cc:3037] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:37.119045 5479 raft_consensus.cc:504] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1030f92396334ff187711f3a1dcb5a8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38861 } } peers { permanent_uuid: "d7bd5aab8f2749338e077d83cbe2806d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } } peers { permanent_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34573 } }
I20260620 14:24:37.119185 5479 leader_election.cc:288] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [CANDIDATE]: Term 1 election: Requested vote from peers 1030f92396334ff187711f3a1dcb5a8d (127.0.98.1:38861), d7bd5aab8f2749338e077d83cbe2806d (127.0.98.3:39509)
I20260620 14:24:37.119309 5151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5946b063c44168b4ce94b1d4b2a10e" candidate_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1030f92396334ff187711f3a1dcb5a8d"
I20260620 14:24:37.119325 5413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5946b063c44168b4ce94b1d4b2a10e" candidate_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bd5aab8f2749338e077d83cbe2806d"
I20260620 14:24:37.119407 5151 raft_consensus.cc:3037] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:37.119403 5413 raft_consensus.cc:3037] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:37.120106 5151 raft_consensus.cc:2454] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 871b9e8c2c6845b2bf090eb5d39d7bd6 in term 1.
I20260620 14:24:37.120146 5413 raft_consensus.cc:2454] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 871b9e8c2c6845b2bf090eb5d39d7bd6 in term 1.
I20260620 14:24:37.120329 5217 leader_election.cc:302] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1030f92396334ff187711f3a1dcb5a8d, 871b9e8c2c6845b2bf090eb5d39d7bd6; no voters:
I20260620 14:24:37.120507 5479 raft_consensus.cc:2781] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:37.120663 5479 raft_consensus.cc:686] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 1 LEADER]: Becoming Leader. State: Replica: 871b9e8c2c6845b2bf090eb5d39d7bd6, State: Running, Role: LEADER
I20260620 14:24:37.120765 5479 consensus_queue.cc:237] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1030f92396334ff187711f3a1dcb5a8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38861 } } peers { permanent_uuid: "d7bd5aab8f2749338e077d83cbe2806d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } } peers { permanent_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34573 } }
I20260620 14:24:37.121634 5008 catalog_manager.cc:5495] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 871b9e8c2c6845b2bf090eb5d39d7bd6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1030f92396334ff187711f3a1dcb5a8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7bd5aab8f2749338e077d83cbe2806d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "871b9e8c2c6845b2bf090eb5d39d7bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34573 } health_report { overall_health: HEALTHY } } }
I20260620 14:24:37.133572 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.132676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.0.98.2:34573)
0620 14:24:37.132734 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.132735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.132742 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:24:37.132775 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.132776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.132785 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.132834 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:37.133312 (+ 478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:37.133313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:37.133313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:37.133317 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:37.133330 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:37.133334 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:37.133441 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:37.133500 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:24:37.133515 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:24:37.135315 5413 raft_consensus.cc:1261] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d [term 1 FOLLOWER]: Refusing update from remote peer 871b9e8c2c6845b2bf090eb5d39d7bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:37.135354 5151 raft_consensus.cc:1261] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d [term 1 FOLLOWER]: Refusing update from remote peer 871b9e8c2c6845b2bf090eb5d39d7bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:37.135705 5483 consensus_queue.cc:1035] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7bd5aab8f2749338e077d83cbe2806d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:37.135847 5479 consensus_queue.cc:1035] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1030f92396334ff187711f3a1dcb5a8d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:37.139153 5488 mvcc.cc:205] Tried to move back new op lower bound from 7298930594343268352 to 7298930594287161344. Current Snapshot: MvccSnapshot[applied={T|T < 7298930594343268352}]
I20260620 14:24:37.140600 5009 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:37.140722 5009 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:37.141456 5009 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cda5982365c149fea9d9cd33168ff56c: Sending DeleteTablet for 3 replicas of tablet cd5946b063c44168b4ce94b1d4b2a10e
I20260620 14:24:37.141800 5131 tablet_service.cc:1514] Processing DeleteTablet for tablet cd5946b063c44168b4ce94b1d4b2a10e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:37 UTC) from {username='slave'} at 127.0.0.1:51784
I20260620 14:24:37.142019 5495 tablet_replica.cc:330] T cd5946b063c44168b4ce94b1d4b2a10e P 1030f92396334ff187711f3a1dcb5a8d: stopping tablet replica
I20260620 14:24:37.142128 5262 tablet_service.cc:1514] Processing DeleteTablet for tablet cd5946b063c44168b4ce94b1d4b2a10e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:37 UTC) from {username='slave'} at 127.0.0.1:55608
I20260620 14:24:37.142158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 5069
I20260620 14:24:37.142968 5393 tablet_service.cc:1514] Processing DeleteTablet for tablet cd5946b063c44168b4ce94b1d4b2a10e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:37 UTC) from {username='slave'} at 127.0.0.1:50912
I20260620 14:24:37.143188 5497 tablet_replica.cc:330] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d: stopping tablet replica
I20260620 14:24:37.143340 5497 raft_consensus.cc:2229] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:37.143501 5497 raft_consensus.cc:2258] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:37.143956 5496 tablet_replica.cc:330] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6: stopping tablet replica
I20260620 14:24:37.143962 5497 ts_tablet_manager.cc:1905] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:37.144052 5496 raft_consensus.cc:2229] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:37.144142 5496 raft_consensus.cc:2258] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:37.144508 5496 ts_tablet_manager.cc:1905] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:37.145849 5497 ts_tablet_manager.cc:1918] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:37.145972 5497 log.cc:1196] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/cd5946b063c44168b4ce94b1d4b2a10e
I20260620 14:24:37.146198 5496 ts_tablet_manager.cc:1918] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:37.146270 5496 log.cc:1196] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/cd5946b063c44168b4ce94b1d4b2a10e
I20260620 14:24:37.146440 5497 ts_tablet_manager.cc:1939] T cd5946b063c44168b4ce94b1d4b2a10e P d7bd5aab8f2749338e077d83cbe2806d: Deleting consensus metadata
I20260620 14:24:37.146559 5496 ts_tablet_manager.cc:1939] T cd5946b063c44168b4ce94b1d4b2a10e P 871b9e8c2c6845b2bf090eb5d39d7bd6: Deleting consensus metadata
I20260620 14:24:37.146900 4995 catalog_manager.cc:4841] TS 871b9e8c2c6845b2bf090eb5d39d7bd6 (127.0.98.2:34573): tablet cd5946b063c44168b4ce94b1d4b2a10e (table test-table [id=47169549d84241aa92395276c2f15003]) successfully deleted
I20260620 14:24:37.146900 4993 catalog_manager.cc:4841] TS d7bd5aab8f2749338e077d83cbe2806d (127.0.98.3:39509): tablet cd5946b063c44168b4ce94b1d4b2a10e (table test-table [id=47169549d84241aa92395276c2f15003]) successfully deleted
W20260620 14:24:37.148023 4995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38861 (error 108)
I20260620 14:24:37.148483 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.148360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38861 (local address 127.0.0.1:51790)
0620 14:24:37.148412 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:37.148450 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:24:37.148574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 5200
W20260620 14:24:37.148639 4995 catalog_manager.cc:4584] TS 1030f92396334ff187711f3a1dcb5a8d (127.0.98.1:38861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd5946b063c44168b4ce94b1d4b2a10e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38861: connect: Connection refused (error 111)
I20260620 14:24:37.154124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 5331
I20260620 14:24:37.158473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 4978
2026-06-20T14:24:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (663 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 14:24:37.178010 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:24:37.179391 392 webserver.cc:460] Webserver started at https://127.0.0.1:45025/ using document root <none> and password file <none>
I20260620 14:24:37.179509 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:24:37.179705 392 webserver.cc:460] Webserver started at http://127.0.0.1:43347/ using document root <none> and password file <none>
2026-06-20T14:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:37Z Disabled control of system clock
I20260620 14:24:37.187496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43155
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43155
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45025/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/testchainca.cert with env {}
W20260620 14:24:37.266357 5504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:37.266570 5504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:37.266592 5504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:37.267750 5504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:37.267791 5504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:37.267803 5504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:37.267817 5504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:37.267828 5504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:24:37.267840 5504 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:24:37.267891 5504 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45025/jwks.json
I20260620 14:24:37.269375 5504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45025/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:37.269626 5504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:37.269843 5504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:37.274667 5509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:37.274909 5512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:37.274915 5504 server_base.cc:1034] running on GCE node
W20260620 14:24:37.274923 5510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:37.275252 5504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:37.275545 5504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:37.276726 5504 hybrid_clock.cc:648] HybridClock initialized: now 1781965477276712 us; error 29 us; skew 500 ppm
I20260620 14:24:37.278044 5504 webserver.cc:460] Webserver started at http://127.0.98.62:33209/ using document root <none> and password file <none>
I20260620 14:24:37.278292 5504 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:37.278386 5504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:37.278527 5504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:37.279551 5504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "72c232c72ddc4a3e9c95b84743c03146"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "8dd193dbdbd82016f7c21f0921aafcb5"
server_key_iv: "b322c9cc27b966e30c6f404b791ce2e5"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.279914 5504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "72c232c72ddc4a3e9c95b84743c03146"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "8dd193dbdbd82016f7c21f0921aafcb5"
server_key_iv: "b322c9cc27b966e30c6f404b791ce2e5"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.281267 5504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:37.282038 5518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:37.282229 5504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:37.282362 5504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "72c232c72ddc4a3e9c95b84743c03146"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "8dd193dbdbd82016f7c21f0921aafcb5"
server_key_iv: "b322c9cc27b966e30c6f404b791ce2e5"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.282449 5504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:37.292858 5504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:37.293170 5504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:37.293325 5504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:37.297286 5504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43155
I20260620 14:24:37.297319 5570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43155 every 8 connection(s)
I20260620 14:24:37.297664 5504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:37.298730 5571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:37.301106 5571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146: Bootstrap starting.
I20260620 14:24:37.301716 5571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:37.301708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 5504
I20260620 14:24:37.301842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:37.302037 392 external_mini_cluster.cc:1428] Setting key a7fbb9f1f1f20a3cdde835230b80d69f
I20260620 14:24:37.302075 5571 log.cc:824] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:37.302914 5571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146: No bootstrap required, opened a new log
I20260620 14:24:37.305262 5571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c232c72ddc4a3e9c95b84743c03146" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43155 } }
I20260620 14:24:37.305426 5571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:37.305457 5571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c232c72ddc4a3e9c95b84743c03146, State: Initialized, Role: FOLLOWER
I20260620 14:24:37.305567 5571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c232c72ddc4a3e9c95b84743c03146" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43155 } }
I20260620 14:24:37.305640 5571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:37.305670 5571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:37.305719 5571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:37.306268 5571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c232c72ddc4a3e9c95b84743c03146" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43155 } }
I20260620 14:24:37.306413 5571 leader_election.cc:302] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72c232c72ddc4a3e9c95b84743c03146; no voters:
I20260620 14:24:37.306607 5571 leader_election.cc:288] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:37.306777 5576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:37.306938 5571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:37.306932 5576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [term 1 LEADER]: Becoming Leader. State: Replica: 72c232c72ddc4a3e9c95b84743c03146, State: Running, Role: LEADER
I20260620 14:24:37.307222 5576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c232c72ddc4a3e9c95b84743c03146" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43155 } }
I20260620 14:24:37.307765 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72c232c72ddc4a3e9c95b84743c03146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c232c72ddc4a3e9c95b84743c03146" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43155 } } }
I20260620 14:24:37.307864 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:37.308127 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.302639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.0.98.62:43155)
0620 14:24:37.302851 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.302859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.302883 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:24:37.303337 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.303344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.303401 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.303515 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:37.303770 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.304616 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.305777 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.306196 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.306300 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:37.307148 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:37.307152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:37.307152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:37.307352 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:37.307392 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:37.307407 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:37.307473 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:37.307571 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:24:37.307652 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260620 14:24:37.308578 5577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72c232c72ddc4a3e9c95b84743c03146. Latest consensus state: current_term: 1 leader_uuid: "72c232c72ddc4a3e9c95b84743c03146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c232c72ddc4a3e9c95b84743c03146" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43155 } } }
I20260620 14:24:37.308652 5577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146 [sys.catalog]: This master's current role is: LEADER
W20260620 14:24:37.308938 5592 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:37.308995 5592 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:37.309077 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:37.309113 5586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:37.309286 5586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:37.310806 5586 catalog_manager.cc:1347] Generated new cluster ID: a02df0f052f747e3b208e61e31ea266c
I20260620 14:24:37.310863 5586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:37.333881 5586 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:37.334692 5586 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:37.344200 5586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72c232c72ddc4a3e9c95b84743c03146: Generated new TSK 0
I20260620 14:24:37.344403 5586 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:24:37.370951 5601 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 14:24:37.371068 5601 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:24:37.371341 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.356003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57290 (local address 127.0.98.62:43155)
0620 14:24:37.356155 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.356160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.356302 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:24:37.356400 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.356401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.356411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.356474 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:24:37.356662 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.357142 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.358269 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.358512 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.358563 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:37.358803 (+ 240us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:24:37.371085 (+ 12282us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45025/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:24:37.371270 (+ 185us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45025/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 14:24:37.371418 5575 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45025/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 14:24:37.371665 5600 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:37.355945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43155 (local address 127.0.0.1:57290)
0620 14:24:37.356198 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:37.356211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:37.356291 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:37.356500 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:37.356501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:37.356503 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:24:37.356567 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:37.356569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.357246 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.357247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:37.358179 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:37.358182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.358580 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.358584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:37.358651 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:37.358680 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:24:37.371259 (+ 12579us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45025/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:24:37.371365 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43155: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45025/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":147,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:43155: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45025/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 14:24:37.372107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 5504
2026-06-20T14:24:37Z chronyd exiting
I20260620 14:24:37.407598 392 test_util.cc:175] -----------------------------------------------
I20260620 14:24:37.407677 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965454096380-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 14:24:37.408430 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:24:37.409145 392 webserver.cc:460] Webserver started at https://127.0.0.1:33383/ using document root <none> and password file <none>
I20260620 14:24:37.409185 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:24:37.409396 392 webserver.cc:460] Webserver started at http://127.0.0.1:41289/ using document root <none> and password file <none>
2026-06-20T14:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:37Z Disabled control of system clock
I20260620 14:24:37.417387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35215
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35215
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33383/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/testchainca.cert with env {}
W20260620 14:24:37.496337 5607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:37.496591 5607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:37.496630 5607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:37.497812 5607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:37.497884 5607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:37.497906 5607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:37.497936 5607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:37.497968 5607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:24:37.497994 5607 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:24:37.498024 5607 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33383/jwks.json
I20260620 14:24:37.499539 5607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33383/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:37.499814 5607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:37.500031 5607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:37.504943 5612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:37.504984 5615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:37.504956 5613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:37.505127 5607 server_base.cc:1034] running on GCE node
I20260620 14:24:37.505477 5607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:37.505805 5607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:37.506971 5607 hybrid_clock.cc:648] HybridClock initialized: now 1781965477506958 us; error 45 us; skew 500 ppm
I20260620 14:24:37.508158 5607 webserver.cc:460] Webserver started at http://127.0.98.62:36267/ using document root <none> and password file <none>
I20260620 14:24:37.508419 5607 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:37.508492 5607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:37.508643 5607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:37.509713 5607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "e23509251a0d401a85762f09d2b0ca9b"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "780d59634deebd00b73a7d76d3a7395c"
server_key_iv: "63a7f0e888131355dfe2e9528b3ce458"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.510094 5607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "e23509251a0d401a85762f09d2b0ca9b"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "780d59634deebd00b73a7d76d3a7395c"
server_key_iv: "63a7f0e888131355dfe2e9528b3ce458"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.511551 5607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:37.512288 5621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:37.512460 5607 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:37.512559 5607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "e23509251a0d401a85762f09d2b0ca9b"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "780d59634deebd00b73a7d76d3a7395c"
server_key_iv: "63a7f0e888131355dfe2e9528b3ce458"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.512642 5607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:37.537986 5607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:37.538399 5607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:37.538575 5607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:37.542770 5607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35215
I20260620 14:24:37.542807 5673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35215 every 8 connection(s)
I20260620 14:24:37.543174 5607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:37.544188 5674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:37.546482 5674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b: Bootstrap starting.
I20260620 14:24:37.547029 5674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:37.547266 5674 log.cc:824] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:37.547926 5674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b: No bootstrap required, opened a new log
I20260620 14:24:37.549422 5674 raft_consensus.cc:348] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23509251a0d401a85762f09d2b0ca9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35215 } }
I20260620 14:24:37.549558 5674 raft_consensus.cc:374] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:37.549583 5674 raft_consensus.cc:729] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23509251a0d401a85762f09d2b0ca9b, State: Initialized, Role: FOLLOWER
I20260620 14:24:37.549723 5674 consensus_queue.cc:260] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23509251a0d401a85762f09d2b0ca9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35215 } }
I20260620 14:24:37.549803 5674 raft_consensus.cc:388] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:37.549831 5674 raft_consensus.cc:482] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:37.549860 5674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:37.550408 5674 raft_consensus.cc:504] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23509251a0d401a85762f09d2b0ca9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35215 } }
I20260620 14:24:37.550516 5674 leader_election.cc:302] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e23509251a0d401a85762f09d2b0ca9b; no voters:
I20260620 14:24:37.550668 5674 leader_election.cc:288] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:37.550696 5677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:37.550813 5677 raft_consensus.cc:686] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [term 1 LEADER]: Becoming Leader. State: Replica: e23509251a0d401a85762f09d2b0ca9b, State: Running, Role: LEADER
I20260620 14:24:37.550906 5677 consensus_queue.cc:237] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23509251a0d401a85762f09d2b0ca9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35215 } }
I20260620 14:24:37.550938 5674 sys_catalog.cc:564] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:37.551259 5679 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e23509251a0d401a85762f09d2b0ca9b. Latest consensus state: current_term: 1 leader_uuid: "e23509251a0d401a85762f09d2b0ca9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23509251a0d401a85762f09d2b0ca9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35215 } } }
I20260620 14:24:37.551355 5679 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:37.551638 5678 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e23509251a0d401a85762f09d2b0ca9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23509251a0d401a85762f09d2b0ca9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35215 } } }
I20260620 14:24:37.551705 5678 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:37.551738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 5607
I20260620 14:24:37.551831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:37.552042 392 external_mini_cluster.cc:1428] Setting key 5227734967c4972a9d10575cf98d1376
I20260620 14:24:37.552035 5683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:37.552663 5683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:37.554461 5683 catalog_manager.cc:1347] Generated new cluster ID: 793b447a082345678dbf97c6ffbaa39a
I20260620 14:24:37.554526 5683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:37.558876 5695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.552520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.0.98.62:35215)
0620 14:24:37.552791 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.552802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.553496 (+ 694us) server_negotiation.cc:407] Connection header received
0620 14:24:37.553627 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.553634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.553715 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.553886 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:37.554098 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.555481 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.556581 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.557274 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.557419 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:37.558115 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:37.558119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:37.558120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:37.558358 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:37.558402 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:37.558417 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:37.558482 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:37.558561 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 14:24:37.558647 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260620 14:24:37.559803 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:37.565284 5683 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:37.565860 5683 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:37.569209 5683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e23509251a0d401a85762f09d2b0ca9b: Generated new TSK 0
I20260620 14:24:37.569380 5683 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:24:37.589231 5704 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 14:24:37.589337 5704 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:24:37.589665 5695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.575151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39332 (local address 127.0.98.62:35215)
0620 14:24:37.575196 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.575198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.575301 (+ 103us) server_negotiation.cc:407] Connection header received
0620 14:24:37.575348 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.575349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.575361 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.575444 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:24:37.575564 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.576385 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.577156 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.577483 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.577557 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:37.577719 (+ 162us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:24:37.589431 (+ 11712us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:24:37.589591 (+ 160us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39332: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 14:24:37.589732 5695 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39332: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 14:24:37.589741 5703 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:37.575060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35215 (local address 127.0.0.1:39332)
0620 14:24:37.575211 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:37.575222 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:37.575296 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:37.575447 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:37.575447 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:37.575449 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:24:37.575497 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:37.575499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.576494 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.576495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:37.577070 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:37.577072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.577558 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.577562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:37.577610 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:37.577615 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:24:37.589550 (+ 11935us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:24:37.589652 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35215: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":106,"threads_started":1}
I20260620 14:24:37.590037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 5607
2026-06-20T14:24:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 14:24:37.627455 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:24:37.628110 392 webserver.cc:460] Webserver started at https://127.0.0.1:45763/ using document root <none> and password file <none>
I20260620 14:24:37.628150 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:24:37.628288 392 webserver.cc:460] Webserver started at http://127.0.0.1:36943/ using document root <none> and password file <none>
2026-06-20T14:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:37Z Disabled control of system clock
I20260620 14:24:37.636833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36777
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36777
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45763/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:24:37.716847 5710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:37.717149 5710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:37.717196 5710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:37.718803 5710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:37.718869 5710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:37.718891 5710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:37.718924 5710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:37.718950 5710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:24:37.718978 5710 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:24:37.719002 5710 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45763/jwks.json
I20260620 14:24:37.720374 5710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45763/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:37.720650 5710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:37.720943 5710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:37.725736 5716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:37.725750 5718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:37.725831 5710 server_base.cc:1034] running on GCE node
W20260620 14:24:37.725750 5715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:37.726186 5710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:37.726519 5710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:37.727669 5710 hybrid_clock.cc:648] HybridClock initialized: now 1781965477727655 us; error 28 us; skew 500 ppm
I20260620 14:24:37.728953 5710 webserver.cc:460] Webserver started at http://127.0.98.62:37929/ using document root <none> and password file <none>
I20260620 14:24:37.729187 5710 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:37.729231 5710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:37.729408 5710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:37.730576 5710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "b3f40beca2dd48bc95019bb4dc9e602c"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "69246aed956b6544563a59e63289c4a3"
server_key_iv: "cc568d11b49c8f2bc717bd69969d0875"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.730954 5710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "b3f40beca2dd48bc95019bb4dc9e602c"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "69246aed956b6544563a59e63289c4a3"
server_key_iv: "cc568d11b49c8f2bc717bd69969d0875"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.732208 5710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:24:37.732870 5724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:37.733036 5710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:37.733141 5710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "b3f40beca2dd48bc95019bb4dc9e602c"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "69246aed956b6544563a59e63289c4a3"
server_key_iv: "cc568d11b49c8f2bc717bd69969d0875"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.733215 5710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:37.755825 5710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:37.756157 5710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:37.756319 5710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:37.760205 5710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36777
I20260620 14:24:37.760237 5776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36777 every 8 connection(s)
I20260620 14:24:37.760583 5710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:37.761299 5777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:37.761323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 5710
I20260620 14:24:37.761413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:37.761620 392 external_mini_cluster.cc:1428] Setting key 430e40c7bf414f6e7c1073cc18a3ee89
I20260620 14:24:37.764496 5777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c: Bootstrap starting.
I20260620 14:24:37.765058 5777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:37.765336 5777 log.cc:824] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:37.766067 5777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c: No bootstrap required, opened a new log
I20260620 14:24:37.767055 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.762149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.0.98.62:36777)
0620 14:24:37.762388 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.762398 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.762432 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:24:37.762844 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.762850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.762910 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.763060 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:37.763288 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.764137 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.765035 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.765543 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.765685 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:37.766467 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:37.766476 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:37.766477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:37.766634 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:37.766667 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:37.766678 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:37.766755 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:37.766812 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 14:24:37.766890 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260620 14:24:37.768100 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:37.768241 5777 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f40beca2dd48bc95019bb4dc9e602c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260620 14:24:37.768397 5777 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:37.768445 5777 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f40beca2dd48bc95019bb4dc9e602c, State: Initialized, Role: FOLLOWER
I20260620 14:24:37.768560 5777 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f40beca2dd48bc95019bb4dc9e602c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260620 14:24:37.768647 5777 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:37.768690 5777 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:37.768739 5777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:37.769404 5777 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f40beca2dd48bc95019bb4dc9e602c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260620 14:24:37.769490 5777 leader_election.cc:302] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3f40beca2dd48bc95019bb4dc9e602c; no voters:
I20260620 14:24:37.769626 5777 leader_election.cc:288] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:37.769672 5782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:37.769797 5782 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [term 1 LEADER]: Becoming Leader. State: Replica: b3f40beca2dd48bc95019bb4dc9e602c, State: Running, Role: LEADER
I20260620 14:24:37.769896 5777 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:37.769889 5782 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f40beca2dd48bc95019bb4dc9e602c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260620 14:24:37.770289 5784 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f40beca2dd48bc95019bb4dc9e602c. Latest consensus state: current_term: 1 leader_uuid: "b3f40beca2dd48bc95019bb4dc9e602c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f40beca2dd48bc95019bb4dc9e602c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } } }
I20260620 14:24:37.770294 5783 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3f40beca2dd48bc95019bb4dc9e602c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f40beca2dd48bc95019bb4dc9e602c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } } }
I20260620 14:24:37.770455 5784 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:37.770465 5783 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c [sys.catalog]: This master's current role is: LEADER
W20260620 14:24:37.771770 5798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:37.771826 5798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:37.771893 5790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:37.772073 5790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:37.773451 5790 catalog_manager.cc:1347] Generated new cluster ID: 863a6d25c5954ce1b6f722a9eb93eec4
I20260620 14:24:37.773511 5790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:37.791287 5790 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:37.791806 5790 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:37.798606 5790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3f40beca2dd48bc95019bb4dc9e602c: Generated new TSK 0
I20260620 14:24:37.798771 5790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:37.812758 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.808815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.0.98.62:36777)
0620 14:24:37.808884 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.808886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.808952 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:24:37.809019 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.809020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.809033 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.809109 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:24:37.809230 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.810260 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.811057 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.811363 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.811439 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:37.811561 (+ 122us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:24:37.812539 (+ 978us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45763/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:24:37.812674 (+ 135us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45763/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 14:24:37.812829 5780 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45763/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 14:24:37.814710 5806 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:37.808731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.0.1:57762)
0620 14:24:37.808892 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:37.808902 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:37.808964 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:37.809123 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:37.809123 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:37.809125 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:24:37.809174 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:37.809176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.810379 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.810379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:37.810974 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:37.810976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.811435 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.811442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:37.811481 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:37.811485 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:24:37.812641 (+ 1156us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45763/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:24:37.814605 (+ 1964us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36777: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45763/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":1752,"thread_start_us":101,"threads_started":1}
I20260620 14:24:37.814939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 5710
E20260620 14:24:37.825459 5807 webserver.cc:533] Webserver: sslize failed:
E20260620 14:24:37.825528 5807 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T14:24:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T14:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:37Z Disabled control of system clock
I20260620 14:24:37.862461 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34109
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34109
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/testchainca.cert with env {}
W20260620 14:24:37.942821 5811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:37.943022 5811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:37.943043 5811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:37.944125 5811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:37.944159 5811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:37.944172 5811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:37.944185 5811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:37.944197 5811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:24:37.944209 5811 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:24:37.944226 5811 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 14:24:37.945607 5811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:37.945797 5811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:37.945950 5811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:37.950560 5819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:37.950551 5817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:37.950539 5816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:37.950760 5811 server_base.cc:1034] running on GCE node
I20260620 14:24:37.950965 5811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:37.951223 5811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:37.952368 5811 hybrid_clock.cc:648] HybridClock initialized: now 1781965477952357 us; error 27 us; skew 500 ppm
I20260620 14:24:37.953531 5811 webserver.cc:460] Webserver started at http://127.0.98.62:37795/ using document root <none> and password file <none>
I20260620 14:24:37.953821 5811 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:37.953871 5811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:37.954042 5811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:37.955137 5811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "05bfe3bbbddd4dcb8fadd569a58dc52f"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "9be264195d7f047a8bf006bdd52fa76d"
server_key_iv: "e7f0f7f904a2add9fbfb5b9c8a978e7e"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.955505 5811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "05bfe3bbbddd4dcb8fadd569a58dc52f"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "9be264195d7f047a8bf006bdd52fa76d"
server_key_iv: "e7f0f7f904a2add9fbfb5b9c8a978e7e"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.956867 5811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:24:37.957567 5825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:37.957734 5811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:37.957831 5811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "05bfe3bbbddd4dcb8fadd569a58dc52f"
format_stamp: "Formatted at 2026-06-20 14:24:37 on dist-test-slave-hfc3"
server_key: "9be264195d7f047a8bf006bdd52fa76d"
server_key_iv: "e7f0f7f904a2add9fbfb5b9c8a978e7e"
server_key_version: "encryptionkey@0"
I20260620 14:24:37.957912 5811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:37.973664 5811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:37.973996 5811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:37.974151 5811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:37.978046 5811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34109
I20260620 14:24:37.978082 5877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34109 every 8 connection(s)
I20260620 14:24:37.978514 5811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:37.979417 5878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:37.981568 5878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f: Bootstrap starting.
I20260620 14:24:37.982086 5878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:37.982313 5878 log.cc:824] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:37.982995 5878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f: No bootstrap required, opened a new log
I20260620 14:24:37.984480 5878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bfe3bbbddd4dcb8fadd569a58dc52f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } }
I20260620 14:24:37.984609 5878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:37.984632 5878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bfe3bbbddd4dcb8fadd569a58dc52f, State: Initialized, Role: FOLLOWER
I20260620 14:24:37.984728 5878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bfe3bbbddd4dcb8fadd569a58dc52f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } }
I20260620 14:24:37.984778 5878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:37.984797 5878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:37.984823 5878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:37.985276 5878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bfe3bbbddd4dcb8fadd569a58dc52f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } }
I20260620 14:24:37.985355 5878 leader_election.cc:302] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05bfe3bbbddd4dcb8fadd569a58dc52f; no voters:
I20260620 14:24:37.985474 5878 leader_election.cc:288] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:37.985515 5881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:37.985663 5881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [term 1 LEADER]: Becoming Leader. State: Replica: 05bfe3bbbddd4dcb8fadd569a58dc52f, State: Running, Role: LEADER
I20260620 14:24:37.985724 5878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:37.985751 5881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bfe3bbbddd4dcb8fadd569a58dc52f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } }
I20260620 14:24:37.986168 5882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05bfe3bbbddd4dcb8fadd569a58dc52f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bfe3bbbddd4dcb8fadd569a58dc52f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } } }
I20260620 14:24:37.986264 5882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:37.986387 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05bfe3bbbddd4dcb8fadd569a58dc52f. Latest consensus state: current_term: 1 leader_uuid: "05bfe3bbbddd4dcb8fadd569a58dc52f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bfe3bbbddd4dcb8fadd569a58dc52f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } } }
I20260620 14:24:37.986472 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:37.986516 5889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:37.986763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 5811
I20260620 14:24:37.986956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:37.987013 5889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:37.987197 392 external_mini_cluster.cc:1428] Setting key b1c84e3377552e50a1da2c97ff058d47
I20260620 14:24:37.989557 5889 catalog_manager.cc:1347] Generated new cluster ID: 9e2da44b99e547deb62b9095cedc0837
I20260620 14:24:37.989609 5889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:37.993165 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:37.987652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.0.98.62:34109)
0620 14:24:37.988095 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:37.988105 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:37.988133 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:24:37.988242 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:37.988250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:37.988315 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:37.988476 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:37.988680 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.989927 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.991057 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:37.991649 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:37.991757 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:37.992488 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:37.992492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:37.992492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:24:37.992686 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:37.992724 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:24:37.992737 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:37.992801 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:37.992904 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:24:37.992975 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":97,"threads_started":1}
I20260620 14:24:37.994010 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:24:38.004305 5889 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:38.004992 5889 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:38.016135 5889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05bfe3bbbddd4dcb8fadd569a58dc52f: Generated new TSK 0
I20260620 14:24:38.016347 5889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:38.056053 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:38.028095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.0.98.62:34109)
0620 14:24:38.028143 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:38.028145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:38.028246 (+ 101us) server_negotiation.cc:407] Connection header received
0620 14:24:38.028306 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:38.028307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:38.028317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:38.028389 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:24:38.028534 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:38.029035 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:38.030169 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:38.030432 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:38.030485 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:38.030685 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:24:38.055770 (+ 25085us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:24:38.055973 (+ 203us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53438: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 14:24:38.056135 5899 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53438: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 14:24:38.056150 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:38.027986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34109 (local address 127.0.0.1:53438)
0620 14:24:38.028163 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:38.028174 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:38.028243 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:38.028399 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:38.028400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:38.028401 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:24:38.028465 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:38.028467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:38.029115 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:38.029116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:38.030078 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:38.030082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:38.030511 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:38.030518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:38.030582 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:38.030588 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:24:38.055921 (+ 25333us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:24:38.056037 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34109: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":113,"threads_started":1}
I20260620 14:24:38.056447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 5811
2026-06-20T14:24:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (219 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (96 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (90 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:38 dist-test-slave-hfc3 krb5kdc[5919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:38 dist-test-slave-hfc3 krb5kdc[5919](info): set up 2 sockets
Jun 20 14:24:38 dist-test-slave-hfc3 krb5kdc[5919](info): commencing operation
krb5kdc: starting...
W20260620 14:24:39.270727 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:39 dist-test-slave-hfc3 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:39.391145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34053
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:39.470732 5932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:39.470955 5932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:39.470988 5932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:39.472112 5932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:39.472178 5932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:39.472203 5932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:39.472232 5932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:39.472265 5932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:39.473779 5932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:39.474045 5932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:39.474268 5932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:39.479111 5940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:39.479107 5937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:39.479295 5938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:39.479357 5932 server_base.cc:1034] running on GCE node
I20260620 14:24:39.479627 5932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:39.479918 5932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:39.481086 5932 hybrid_clock.cc:648] HybridClock initialized: now 1781965479481075 us; error 44 us; skew 500 ppm
Jun 20 14:24:39 dist-test-slave-hfc3 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:39.483518 5932 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:39.483943 5932 webserver.cc:460] Webserver started at http://127.0.98.62:35243/ using document root <none> and password file <none>
I20260620 14:24:39.484195 5932 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:39.484266 5932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:39.484411 5932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:39.485467 5932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "a5de3e7ca2a14021bcce46382bd52b81"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-hfc3"
server_key: "5df5c906c28d991c6c56020ef8b453b9"
server_key_iv: "2b8f93af36e3e17b878aae0cf827f8f0"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.485852 5932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5de3e7ca2a14021bcce46382bd52b81"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-hfc3"
server_key: "5df5c906c28d991c6c56020ef8b453b9"
server_key_iv: "2b8f93af36e3e17b878aae0cf827f8f0"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.487193 5932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:39.487824 5947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:39.487989 5932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:39.488099 5932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "a5de3e7ca2a14021bcce46382bd52b81"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-hfc3"
server_key: "5df5c906c28d991c6c56020ef8b453b9"
server_key_iv: "2b8f93af36e3e17b878aae0cf827f8f0"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.488171 5932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:39.501300 5932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:39.503262 5932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:39.503432 5932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:39.507380 5932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34053
I20260620 14:24:39.507431 5999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34053 every 8 connection(s)
I20260620 14:24:39.507962 5932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:39.508600 6000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:39.510931 6000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81: Bootstrap starting.
I20260620 14:24:39.511475 6000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:39.511711 6000 log.cc:824] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:39.512362 6000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81: No bootstrap required, opened a new log
I20260620 14:24:39.513849 6000 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5de3e7ca2a14021bcce46382bd52b81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34053 } }
I20260620 14:24:39.513993 6000 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:39.514016 6000 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5de3e7ca2a14021bcce46382bd52b81, State: Initialized, Role: FOLLOWER
I20260620 14:24:39.514153 6000 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5de3e7ca2a14021bcce46382bd52b81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34053 } }
I20260620 14:24:39.514241 6000 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:39.514266 6000 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:39.514294 6000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:39.515185 6000 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5de3e7ca2a14021bcce46382bd52b81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34053 } }
I20260620 14:24:39.515313 6000 leader_election.cc:302] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5de3e7ca2a14021bcce46382bd52b81; no voters:
I20260620 14:24:39.515443 6000 leader_election.cc:288] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:39.515522 6003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:39.515705 6003 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [term 1 LEADER]: Becoming Leader. State: Replica: a5de3e7ca2a14021bcce46382bd52b81, State: Running, Role: LEADER
I20260620 14:24:39.515726 6000 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:39.515794 6003 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5de3e7ca2a14021bcce46382bd52b81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34053 } }
I20260620 14:24:39.516233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 5932
I20260620 14:24:39.516363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:39.516621 392 external_mini_cluster.cc:1428] Setting key 77dfe32ce8a7b336467c2824d29e7993
I20260620 14:24:39.516753 6004 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5de3e7ca2a14021bcce46382bd52b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5de3e7ca2a14021bcce46382bd52b81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34053 } } }
I20260620 14:24:39.516834 6004 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:39.517001 6005 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5de3e7ca2a14021bcce46382bd52b81. Latest consensus state: current_term: 1 leader_uuid: "a5de3e7ca2a14021bcce46382bd52b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5de3e7ca2a14021bcce46382bd52b81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34053 } } }
I20260620 14:24:39.517091 6005 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81 [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:39.517421 6016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:39.518113 6016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:39.519634 6016 catalog_manager.cc:1347] Generated new cluster ID: 5a366250f6b14804a6b001a1606d8476
I20260620 14:24:39.519687 6016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:24:39 dist-test-slave-hfc3 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:39.527704 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:39.517310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.0.98.62:34053)
0620 14:24:39.517517 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:39.517527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:39.517557 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:24:39.517679 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:39.517687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:39.517748 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:39.517890 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:39.518645 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:39.519933 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:39.520713 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:39.521154 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:39.521269 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:39.523639 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:39.523643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:39.523644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:39.523655 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:39.526010 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:39.526428 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:39.526432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:39.526432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:39.526522 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:39.526863 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:39.526864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:39.526864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:39.527101 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:39.527280 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:39.527356 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 14:24:39.527497 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260620 14:24:39.535370 6016 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:39.535888 6016 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:39.538429 6016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5de3e7ca2a14021bcce46382bd52b81: Generated new TSK 0
I20260620 14:24:39.538651 6016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:39.571100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34053
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:39.655412 6026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:39.655627 6026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:39.655651 6026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:39.656787 6026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:39.656831 6026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:39.656884 6026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:39.658354 6026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:39.658612 6026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:39.658840 6026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:39.663836 6034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:39.663964 6031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:39.664037 6026 server_base.cc:1034] running on GCE node
W20260620 14:24:39.664129 6032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:39.664326 6026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:39.664579 6026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:39.665743 6026 hybrid_clock.cc:648] HybridClock initialized: now 1781965479665740 us; error 36 us; skew 500 ppm
Jun 20 14:24:39 dist-test-slave-hfc3 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:39.668213 6026 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:39.668642 6026 webserver.cc:460] Webserver started at http://127.0.98.1:46205/ using document root <none> and password file <none>
I20260620 14:24:39.668864 6026 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:39.668931 6026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:39.669080 6026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:39.670125 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "202f15c61e6342eaad03b007f4f4a50c"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-hfc3"
server_key: "415d72401009be7d7339d894dbac9788"
server_key_iv: "7cc2890a78b848c38c443f31ade2f08a"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.670519 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "202f15c61e6342eaad03b007f4f4a50c"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-hfc3"
server_key: "415d72401009be7d7339d894dbac9788"
server_key_iv: "7cc2890a78b848c38c443f31ade2f08a"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.671804 6026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:39.672511 6041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:39.672677 6026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:39.672773 6026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "202f15c61e6342eaad03b007f4f4a50c"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-hfc3"
server_key: "415d72401009be7d7339d894dbac9788"
server_key_iv: "7cc2890a78b848c38c443f31ade2f08a"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.672854 6026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:39.682704 6026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:39.684444 6026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:39.684595 6026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:39.684810 6026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:39.685112 6026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:39.685166 6026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:39.685209 6026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:39.685250 6026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:39.690973 6026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36429
I20260620 14:24:39.690999 6154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36429 every 8 connection(s)
I20260620 14:24:39.691324 6026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:39.696411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 6026
I20260620 14:24:39.696527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:39.696741 392 external_mini_cluster.cc:1428] Setting key 6b77586a3a2394575913f2bef186bda2
Jun 20 14:24:39 dist-test-slave-hfc3 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:39.701187 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:39.692094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59865 (local address 127.0.98.62:34053)
0620 14:24:39.692151 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:39.692153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:39.692398 (+ 245us) server_negotiation.cc:407] Connection header received
0620 14:24:39.693378 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:39.693380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:39.693397 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:39.693450 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:39.694065 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:39.694630 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:39.695626 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:39.695828 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:39.695917 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:39.698248 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:39.698250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:39.698250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:39.698260 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:39.699727 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:39.700231 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:39.700232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:39.700233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:39.700296 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:39.700545 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:39.700546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:39.700547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:39.700662 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:39.700739 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:39.701004 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 14:24:39.701094 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:24:39.701386 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:39.692131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34053 (local address 127.0.98.1:59865)
0620 14:24:39.692282 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:39.692316 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:39.693260 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:39.693522 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:39.693525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:39.693771 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:39.693949 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:39.693953 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:39.694729 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:39.694732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:39.695516 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:39.695518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:39.695939 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:39.695940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:39.696056 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:39.696527 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:39.696537 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:39.698157 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:39.699815 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:39.699822 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:39.699824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:39.700141 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:39.700363 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:39.700363 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:39.700364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:39.700476 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:39.700729 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:39.700736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:39.700874 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 14:24:39.700990 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:24:39.701145 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260620 14:24:39.701861 6155 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34053
I20260620 14:24:39.701951 6155 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:39.702153 6155 heartbeater.cc:511] Master 127.0.98.62:34053 requested a full tablet report, sending...
I20260620 14:24:39.702741 5964 ts_manager.cc:194] Registered new tserver with Master: 202f15c61e6342eaad03b007f4f4a50c (127.0.98.1:36429)
I20260620 14:24:39.703677 5964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59865
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:39.721385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34053
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:39.814957 6160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:39.815160 6160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:39.815183 6160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:39.816344 6160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:39.816382 6160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:39.816407 6160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:39.817946 6160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:39.818184 6160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:39.818456 6160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:39.823354 6166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:39.823421 6168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:39.823434 6165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:39.823479 6160 server_base.cc:1034] running on GCE node
I20260620 14:24:39.823837 6160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:39.824103 6160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:39.825275 6160 hybrid_clock.cc:648] HybridClock initialized: now 1781965479825259 us; error 43 us; skew 500 ppm
Jun 20 14:24:39 dist-test-slave-hfc3 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:39.827893 6160 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:39.828269 6160 webserver.cc:460] Webserver started at http://127.0.98.2:37927/ using document root <none> and password file <none>
I20260620 14:24:39.828469 6160 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:39.828510 6160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:39.828593 6160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:39.829564 6160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "b2ff9e59d14f4660b9b3dccd4be13c14"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-hfc3"
server_key: "dcc0089a922493b443801bad6e4faa52"
server_key_iv: "13f560e41b74eb88f29b418f795aa123"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.829859 6160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b2ff9e59d14f4660b9b3dccd4be13c14"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-hfc3"
server_key: "dcc0089a922493b443801bad6e4faa52"
server_key_iv: "13f560e41b74eb88f29b418f795aa123"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.831161 6160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:39.831877 6175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:39.832062 6160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:39.832149 6160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "b2ff9e59d14f4660b9b3dccd4be13c14"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-hfc3"
server_key: "dcc0089a922493b443801bad6e4faa52"
server_key_iv: "13f560e41b74eb88f29b418f795aa123"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.832223 6160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:39.845290 6160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:39.847146 6160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:39.847294 6160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:39.847508 6160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:39.847826 6160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:39.847862 6160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:39.847908 6160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:39.847935 6160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:39.853631 6160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45205
I20260620 14:24:39.853698 6288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45205 every 8 connection(s)
I20260620 14:24:39.854019 6160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:39.857143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 6160
I20260620 14:24:39.857244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:39.857460 392 external_mini_cluster.cc:1428] Setting key f6ea22b0b80eb99e69aa318744658078
Jun 20 14:24:39 dist-test-slave-hfc3 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:39.864353 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:39.854758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49483 (local address 127.0.98.62:34053)
0620 14:24:39.854801 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:39.854803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:39.854986 (+ 183us) server_negotiation.cc:407] Connection header received
0620 14:24:39.855919 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:39.855920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:39.855930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:39.855990 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:39.856554 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:39.857044 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:39.858547 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:39.858738 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:39.858794 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:39.861435 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:39.861437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:39.861437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:39.861443 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:39.862917 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:39.863419 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:39.863420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:39.863420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:39.863466 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:39.863745 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:39.863745 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:39.863745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:39.863841 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:39.863887 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:39.864228 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 14:24:39.864292 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:24:39.864634 6291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:39.854705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34053 (local address 127.0.98.2:49483)
0620 14:24:39.854861 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:39.854901 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:39.855814 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:39.856060 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:39.856064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:39.856309 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:39.856484 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:39.856488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:39.857132 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:39.857133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:39.858473 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:39.858477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:39.858819 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:39.858820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:39.858964 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:39.859461 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:39.859471 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:39.861338 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:39.863000 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:39.863007 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:39.863009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:39.863343 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:39.863531 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:39.863532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:39.863533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:39.863675 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:39.863911 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:39.863917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:39.864091 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 14:24:39.864213 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:24:39.864380 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260620 14:24:39.865160 6289 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34053
I20260620 14:24:39.865267 6289 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:39.865458 6289 heartbeater.cc:511] Master 127.0.98.62:34053 requested a full tablet report, sending...
I20260620 14:24:39.865824 5964 ts_manager.cc:194] Registered new tserver with Master: b2ff9e59d14f4660b9b3dccd4be13c14 (127.0.98.2:45205)
I20260620 14:24:39.866202 5964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49483
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:39.883296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34053
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:39.964802 6294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:39.964993 6294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:39.965016 6294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:39.966081 6294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:39.966113 6294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:39.966138 6294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:39.967650 6294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:39.967829 6294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:39.967995 6294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:39.972841 6302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:39.972862 6300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:39.973008 6294 server_base.cc:1034] running on GCE node
W20260620 14:24:39.972836 6299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:39.973453 6294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:39.973703 6294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:39.974855 6294 hybrid_clock.cc:648] HybridClock initialized: now 1781965479974854 us; error 35 us; skew 500 ppm
Jun 20 14:24:39 dist-test-slave-hfc3 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:39.977308 6294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:39.977802 6294 webserver.cc:460] Webserver started at http://127.0.98.3:33905/ using document root <none> and password file <none>
I20260620 14:24:39.978035 6294 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:39.978104 6294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:39.978297 6294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:39.979333 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "d7100d3666514ce1a75350be1a9d1db5"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-hfc3"
server_key: "769d27b726739576b71b6061a7f85f2d"
server_key_iv: "97859e2f1f158931e40ba0aac516d14d"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.979692 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d7100d3666514ce1a75350be1a9d1db5"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-hfc3"
server_key: "769d27b726739576b71b6061a7f85f2d"
server_key_iv: "97859e2f1f158931e40ba0aac516d14d"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.980984 6294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:24:39.981674 6309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:39.981840 6294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:39.981935 6294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "d7100d3666514ce1a75350be1a9d1db5"
format_stamp: "Formatted at 2026-06-20 14:24:39 on dist-test-slave-hfc3"
server_key: "769d27b726739576b71b6061a7f85f2d"
server_key_iv: "97859e2f1f158931e40ba0aac516d14d"
server_key_version: "encryptionkey@0"
I20260620 14:24:39.982016 6294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:40.017992 6294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:40.019976 6294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:40.020151 6294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:40.020392 6294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:40.020713 6294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:40.020769 6294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:40.020813 6294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:40.020838 6294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:40.026616 6294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33637
I20260620 14:24:40.026675 6422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33637 every 8 connection(s)
I20260620 14:24:40.027017 6294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:40.028213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 6294
I20260620 14:24:40.028295 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:40.028486 392 external_mini_cluster.cc:1428] Setting key 5cb70d9d0c59bf5c9d314a4b8dd27507
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:40.036859 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:40.027765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57041 (local address 127.0.98.62:34053)
0620 14:24:40.027839 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:40.027841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:40.028100 (+ 259us) server_negotiation.cc:407] Connection header received
0620 14:24:40.029282 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:40.029283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:40.029296 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:40.029353 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:40.030028 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.030698 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.031654 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.031832 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.031884 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:40.034028 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:40.034029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:40.034030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:40.034037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:40.035426 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:40.035853 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:40.035854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:40.035854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:40.035912 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:40.036204 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:40.036205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:40.036205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:40.036324 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:40.036402 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:40.036682 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 14:24:40.036764 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:24:40.037074 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:40.027737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34053 (local address 127.0.98.3:57041)
0620 14:24:40.027946 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:40.027994 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:40.029173 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:40.029431 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:40.029435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:40.029712 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:40.029908 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:40.029912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.030775 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.030776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:40.031580 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:40.031582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.031903 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.031903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:40.032010 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:40.032499 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:40.032507 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:40.033919 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:40.035530 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:40.035537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:40.035539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:40.035771 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:40.036019 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:40.036020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:40.036021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:40.036137 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:40.036392 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:40.036398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:40.036549 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 14:24:40.036659 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:24:40.036823 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
I20260620 14:24:40.037624 6423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34053
I20260620 14:24:40.037744 6423 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:40.037951 6423 heartbeater.cc:511] Master 127.0.98.62:34053 requested a full tablet report, sending...
I20260620 14:24:40.038380 5964 ts_manager.cc:194] Registered new tserver with Master: d7100d3666514ce1a75350be1a9d1db5 (127.0.98.3:33637)
I20260620 14:24:40.038784 5964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57041
I20260620 14:24:40.040115 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/test-admin.keytab.
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:40.055088 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 14:24:40.056768 6430 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:40.055949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.0.98.1:36429)
0620 14:24:40.056131 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:40.056136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:40.056147 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:24:40.056173 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:40.056174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:40.056187 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:40.056253 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:40.056683 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48552: BlockingRecv error: recv got EOF from 127.0.0.1:48552 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
W20260620 14:24:40.056931 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:40.057925 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:40.058907 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:24:40.066537 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:40.059414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.0.98.1:36429)
0620 14:24:40.059474 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:40.059476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:40.059552 (+ 76us) server_negotiation.cc:407] Connection header received
0620 14:24:40.059637 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:40.059638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:40.059655 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:40.059717 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:40.060130 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.060863 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.061605 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.061921 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.062000 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:40.064072 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:40.064079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:40.064079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:40.064085 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:40.065521 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:40.065839 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:40.065842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:40.065843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:40.065894 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:40.066051 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:40.066052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:40.066052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:40.066180 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:40.066287 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:40.066393 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 14:24:40.066465 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 14:24:40.068241 6430 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:40.067789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.0.98.1:36429)
0620 14:24:40.067823 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:40.067825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:40.067839 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:24:40.067873 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:40.067873 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:40.067881 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:40.067927 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:40.068168 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48580: BlockingRecv error: recv got EOF from 127.0.0.1:48580 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 14:24:40.068359 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:40.069324 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:40.070135 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:24:40.077721 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:40.070665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.0.98.1:36429)
0620 14:24:40.070714 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:40.070716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:40.070776 (+ 60us) server_negotiation.cc:407] Connection header received
0620 14:24:40.070846 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:40.070847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:40.070855 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:40.070924 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:40.071282 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.071755 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.072431 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.072636 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.072736 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:40.075086 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:40.075087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:40.075088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:40.075096 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:40.076827 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:40.077127 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:40.077128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:40.077128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:40.077179 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:40.077333 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:40.077334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:40.077334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:40.077432 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:40.077502 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:40.077591 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:24:40.077649 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:24:40.079334 6430 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:40.078851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.0.98.1:36429)
0620 14:24:40.078906 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:40.078908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:40.078943 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:24:40.078990 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:40.078991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:40.078998 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:40.079040 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:40.079285 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48592: BlockingRecv error: recv got EOF from 127.0.0.1:48592 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 14:24:40.079492 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965454096380-392-0/krb5kdc/krb5cc)
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:40.080442 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:40.081257 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:24:40.087863 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:40.081743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.0.98.1:36429)
0620 14:24:40.081836 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:40.081837 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:40.081858 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:24:40.081916 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:40.081917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:40.081925 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:40.081968 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:40.082297 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.082807 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.083439 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:40.083613 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:40.083660 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:40.085608 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:40.085609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:40.085609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:40.085615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:40.086989 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:40.087267 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:40.087267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:40.087268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:40.087314 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:40.087450 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:40.087451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:40.087451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:40.087525 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:40.087588 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:40.087668 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:24:40.087723 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:24:40.088474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 6026
I20260620 14:24:40.093817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 6160
I20260620 14:24:40.098172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 6294
I20260620 14:24:40.102620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 5932
2026-06-20T14:24:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1861 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[6443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[6443](info): set up 2 sockets
Jun 20 14:24:40 dist-test-slave-hfc3 krb5kdc[6443](info): commencing operation
krb5kdc: starting...
W20260620 14:24:41.218698 392 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:41 dist-test-slave-hfc3 krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 14:24:41.343492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34461
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:41.425493 6456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:41.425771 6456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:41.425809 6456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:41.427132 6456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:41.427217 6456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:41.427240 6456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:41.427261 6456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:41.427292 6456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:41.428889 6456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:41.429210 6456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:41.429491 6456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:41.434664 6462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:41.434660 6461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:41.434674 6464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:41.434998 6456 server_base.cc:1034] running on GCE node
I20260620 14:24:41.435386 6456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:41.435662 6456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:41.436810 6456 hybrid_clock.cc:648] HybridClock initialized: now 1781965481436790 us; error 40 us; skew 500 ppm
Jun 20 14:24:41 dist-test-slave-hfc3 krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:41.439429 6456 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 14:24:41.439891 6456 webserver.cc:460] Webserver started at http://127.0.98.62:36277/ using document root <none> and password file <none>
I20260620 14:24:41.440094 6456 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:41.440135 6456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:41.440227 6456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:41.441308 6456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "f42193629d04488ca85d1942981c8cac"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "ef07b69e6a7adade41c3dd7ebb7b202f"
server_key_iv: "24b52422a67168d63dec325300174145"
server_key_version: "encryptionkey@0"
I20260620 14:24:41.441622 6456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "f42193629d04488ca85d1942981c8cac"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "ef07b69e6a7adade41c3dd7ebb7b202f"
server_key_iv: "24b52422a67168d63dec325300174145"
server_key_version: "encryptionkey@0"
I20260620 14:24:41.443001 6456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:24:41.443730 6471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:41.443903 6456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:41.444025 6456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "f42193629d04488ca85d1942981c8cac"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "ef07b69e6a7adade41c3dd7ebb7b202f"
server_key_iv: "24b52422a67168d63dec325300174145"
server_key_version: "encryptionkey@0"
I20260620 14:24:41.444105 6456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:41.467012 6456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:41.469138 6456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:41.469357 6456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:41.473613 6456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34461
I20260620 14:24:41.473651 6523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34461 every 8 connection(s)
I20260620 14:24:41.474038 6456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:41.475097 6524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:41.477312 6524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac: Bootstrap starting.
I20260620 14:24:41.477860 6524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:41.478093 6524 log.cc:824] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:41.478826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 6456
I20260620 14:24:41.478865 6524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac: No bootstrap required, opened a new log
I20260620 14:24:41.478952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:41.479153 392 external_mini_cluster.cc:1428] Setting key c52d9cb44050f0f46be9f75491510a05
I20260620 14:24:41.481504 6524 raft_consensus.cc:348] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } }
I20260620 14:24:41.481704 6524 raft_consensus.cc:374] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:41.481750 6524 raft_consensus.cc:729] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42193629d04488ca85d1942981c8cac, State: Initialized, Role: FOLLOWER
I20260620 14:24:41.481876 6524 consensus_queue.cc:260] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } }
I20260620 14:24:41.481957 6524 raft_consensus.cc:388] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:41.481979 6524 raft_consensus.cc:482] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:41.482031 6524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:41.482647 6524 raft_consensus.cc:504] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } }
I20260620 14:24:41.482767 6524 leader_election.cc:302] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f42193629d04488ca85d1942981c8cac; no voters:
I20260620 14:24:41.482925 6524 leader_election.cc:288] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:41.483019 6529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:41.483198 6529 raft_consensus.cc:686] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 1 LEADER]: Becoming Leader. State: Replica: f42193629d04488ca85d1942981c8cac, State: Running, Role: LEADER
I20260620 14:24:41.483214 6524 sys_catalog.cc:564] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:41.483287 6529 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } }
I20260620 14:24:41.483605 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } }
I20260620 14:24:41.483685 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:41.483935 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42193629d04488ca85d1942981c8cac. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } }
I20260620 14:24:41.484402 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:41.484342 6536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:41.485030 6536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:41.486696 6536 catalog_manager.cc:1347] Generated new cluster ID: 0a89a7f453ca47f8afefe52921a46e36
I20260620 14:24:41.486750 6536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:24:41 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:41.491781 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:41.479738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.0.98.62:34461)
0620 14:24:41.479940 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:41.479949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:41.479983 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:24:41.481450 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:41.481459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:41.481525 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:41.481691 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:41.482649 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.484105 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.485136 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.485562 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.485684 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:41.488290 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:41.488294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:41.488294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:41.488308 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:41.490086 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:41.490567 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:41.490570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:41.490571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:41.490667 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:41.490892 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:41.490893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:41.490893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:41.491139 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:41.491316 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:41.491438 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 14:24:41.491590 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260620 14:24:41.500957 6536 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:41.501539 6536 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:41.511631 6536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac: Generated new TSK 0
I20260620 14:24:41.511864 6536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 14:24:41.537622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34461
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:41.625758 6550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:41.626010 6550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:41.626053 6550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:41.627331 6550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:41.627410 6550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:41.627457 6550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:41.628988 6550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:41.629284 6550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:41.629524 6550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:41.634688 6555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:41.634694 6558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:41.634694 6556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:41.635021 6550 server_base.cc:1034] running on GCE node
I20260620 14:24:41.635277 6550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:41.635538 6550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:41.636737 6550 hybrid_clock.cc:648] HybridClock initialized: now 1781965481636718 us; error 30 us; skew 500 ppm
Jun 20 14:24:41 dist-test-slave-hfc3 krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:41.642477 6550 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:24:41.642910 6550 webserver.cc:460] Webserver started at http://127.0.98.1:37137/ using document root <none> and password file <none>
I20260620 14:24:41.643106 6550 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:41.643147 6550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:41.643237 6550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:41.644240 6550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ff984641c0d4254878dd7c59a393d5d"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "a5d4bfda6f2caf49449ae418947bfe77"
server_key_iv: "bdb8e92765d010276991fd60ffc24569"
server_key_version: "encryptionkey@0"
I20260620 14:24:41.644539 6550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ff984641c0d4254878dd7c59a393d5d"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "a5d4bfda6f2caf49449ae418947bfe77"
server_key_iv: "bdb8e92765d010276991fd60ffc24569"
server_key_version: "encryptionkey@0"
I20260620 14:24:41.645821 6550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:41.646610 6565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:41.646800 6550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:41.646907 6550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "0ff984641c0d4254878dd7c59a393d5d"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "a5d4bfda6f2caf49449ae418947bfe77"
server_key_iv: "bdb8e92765d010276991fd60ffc24569"
server_key_version: "encryptionkey@0"
I20260620 14:24:41.646986 6550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:41.660827 6550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:41.663282 6550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:41.663437 6550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:41.663769 6550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:41.664130 6550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:41.664177 6550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:41.664201 6550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:41.664247 6550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:41.670045 6550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44637
I20260620 14:24:41.670143 6678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44637 every 8 connection(s)
I20260620 14:24:41.670540 6550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:41.673759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 6550
I20260620 14:24:41.673869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:41.674109 392 external_mini_cluster.cc:1428] Setting key 8ffe95f0450685636eb0ce32be51d45d
Jun 20 14:24:41 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:41.680999 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:41.671308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33049 (local address 127.0.98.62:34461)
0620 14:24:41.671355 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:41.671358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:41.671585 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:24:41.672107 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:41.672109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:41.672120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:41.672187 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:41.673161 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.673731 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.675193 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.675404 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.675484 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:41.678012 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:41.678014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:41.678014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:41.678023 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:41.679528 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:41.680006 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:41.680008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:41.680008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:41.680073 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:41.680333 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:41.680334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:41.680334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:41.680454 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:41.680529 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:41.680814 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:24:41.680905 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:24:41.681224 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:41.671269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.1:33049)
0620 14:24:41.671450 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:41.671496 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:41.671993 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:41.672264 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:41.672268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:41.672850 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:41.673046 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:41.673050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.673870 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.673872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:41.675081 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:41.675084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.675493 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.675495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:41.675627 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:41.676269 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:41.676278 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:41.677909 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:41.679678 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:41.679685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:41.679688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:41.679926 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:41.680143 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:41.680144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:41.680144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:41.680263 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:41.680532 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:41.680540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:41.680683 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 14:24:41.680794 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:24:41.680958 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260620 14:24:41.682029 6679 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34461
I20260620 14:24:41.682129 6679 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:41.682438 6679 heartbeater.cc:511] Master 127.0.98.62:34461 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:24:41.683055 6488 ts_manager.cc:194] Registered new tserver with Master: 0ff984641c0d4254878dd7c59a393d5d (127.0.98.1:44637)
I20260620 14:24:41.683770 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 14:24:41.699399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34461
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:41.782425 6684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:41.782691 6684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:41.782732 6684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:41.784173 6684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:41.784238 6684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:41.784286 6684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:41.785748 6684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:41.786016 6684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:41.786289 6684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:41.791448 6684 server_base.cc:1034] running on GCE node
W20260620 14:24:41.791455 6689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:41.791577 6692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:41.791471 6690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:41.791821 6684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:41.792022 6684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:41.793187 6684 hybrid_clock.cc:648] HybridClock initialized: now 1781965481793177 us; error 33 us; skew 500 ppm
Jun 20 14:24:41 dist-test-slave-hfc3 krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:41.795783 6684 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:24:41.796219 6684 webserver.cc:460] Webserver started at http://127.0.98.2:42641/ using document root <none> and password file <none>
I20260620 14:24:41.796440 6684 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:41.796507 6684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:41.796661 6684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:41.797783 6684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "44cafb4598cc46bb94b5495eef807646"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "7317697df9626e6800444d5823bb625c"
server_key_iv: "4fa582a4a39191e4e7a30d4acd9dada8"
server_key_version: "encryptionkey@0"
I20260620 14:24:41.798143 6684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "44cafb4598cc46bb94b5495eef807646"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "7317697df9626e6800444d5823bb625c"
server_key_iv: "4fa582a4a39191e4e7a30d4acd9dada8"
server_key_version: "encryptionkey@0"
I20260620 14:24:41.799542 6684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:41.800374 6699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:41.800572 6684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:41.800716 6684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "44cafb4598cc46bb94b5495eef807646"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "7317697df9626e6800444d5823bb625c"
server_key_iv: "4fa582a4a39191e4e7a30d4acd9dada8"
server_key_version: "encryptionkey@0"
I20260620 14:24:41.800794 6684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:41.813712 6684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:41.815903 6684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:41.816073 6684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:41.816327 6684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:41.816655 6684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:41.816709 6684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:41.816745 6684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:41.816773 6684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:41.822575 6684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35279
I20260620 14:24:41.822736 6812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35279 every 8 connection(s)
I20260620 14:24:41.822995 6684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:41.824679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 6684
I20260620 14:24:41.824793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:41.824983 392 external_mini_cluster.cc:1428] Setting key 593d4357d34844422a6e677209914876
Jun 20 14:24:41 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:24:41.833837 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:41.823803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53135 (local address 127.0.98.62:34461)
0620 14:24:41.823870 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:41.823873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:41.824086 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:24:41.824671 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:41.824673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:41.824685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:41.824760 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:41.826059 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.826919 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.828031 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.828255 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.828337 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:41.830735 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:41.830736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:41.830736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:41.830744 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:41.832378 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:41.832842 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:41.832843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:41.832843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:41.832906 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:41.833162 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:41.833163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:41.833164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:41.833287 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:41.833363 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:41.833663 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 14:24:41.833744 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:24:41.834026 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:41.823785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.2:53135)
0620 14:24:41.823952 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:41.823993 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:41.824519 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:41.824879 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:41.824884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:41.825732 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:41.825961 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:41.825967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.827023 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.827024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:41.827916 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:41.827919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.828343 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.828345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:41.828478 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:41.829070 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:41.829078 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:41.830631 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:41.832495 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:41.832503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:41.832505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:41.832745 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:41.832982 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:41.832983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:41.832983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:41.833086 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:41.833386 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:41.833392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:41.833519 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 14:24:41.833632 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:24:41.833791 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260620 14:24:41.834548 6813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34461
I20260620 14:24:41.834632 6813 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:41.834820 6813 heartbeater.cc:511] Master 127.0.98.62:34461 requested a full tablet report, sending...
I20260620 14:24:41.835301 6488 ts_manager.cc:194] Registered new tserver with Master: 44cafb4598cc46bb94b5495eef807646 (127.0.98.2:35279)
I20260620 14:24:41.835711 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:24:41.849402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34461
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:41.931314 6818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:41.931527 6818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:41.931550 6818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:41.932644 6818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:41.932683 6818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:41.932711 6818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:41.934144 6818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:41.934401 6818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:41.934598 6818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:41.939450 6823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:41.939513 6826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:41.939450 6824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:41.939694 6818 server_base.cc:1034] running on GCE node
I20260620 14:24:41.939901 6818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:41.940166 6818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:41.941336 6818 hybrid_clock.cc:648] HybridClock initialized: now 1781965481941340 us; error 45 us; skew 500 ppm
Jun 20 14:24:41 dist-test-slave-hfc3 krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:41.944062 6818 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:24:41.944566 6818 webserver.cc:460] Webserver started at http://127.0.98.3:37933/ using document root <none> and password file <none>
I20260620 14:24:41.944801 6818 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:41.944870 6818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:41.945030 6818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:41.946149 6818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "4b3a80c21b024326846d3d812178d470"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "a9257d2fd81772f699b4cf43c049a3f1"
server_key_iv: "782c72ee42ed166bd1dec58f37434d62"
server_key_version: "encryptionkey@0"
I20260620 14:24:41.946554 6818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4b3a80c21b024326846d3d812178d470"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "a9257d2fd81772f699b4cf43c049a3f1"
server_key_iv: "782c72ee42ed166bd1dec58f37434d62"
server_key_version: "encryptionkey@0"
I20260620 14:24:41.948053 6818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:41.948974 6833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:41.949257 6818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:41.949373 6818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "4b3a80c21b024326846d3d812178d470"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "a9257d2fd81772f699b4cf43c049a3f1"
server_key_iv: "782c72ee42ed166bd1dec58f37434d62"
server_key_version: "encryptionkey@0"
I20260620 14:24:41.949450 6818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:41.958251 6818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:41.960261 6818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:41.960420 6818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:41.960657 6818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:41.960959 6818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:41.961014 6818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:41.961055 6818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:41.961083 6818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:41.967049 6818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37759
I20260620 14:24:41.967094 6946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37759 every 8 connection(s)
I20260620 14:24:41.967427 6818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:24:41 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:41.974838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 6818
I20260620 14:24:41.974947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:41.975148 392 external_mini_cluster.cc:1428] Setting key 830f5705f23d58dcb39ee569ea6389db
I20260620 14:24:41.977336 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:41.968156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48247 (local address 127.0.98.62:34461)
0620 14:24:41.968203 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:41.968205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:41.968406 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:24:41.968866 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:41.968867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:41.968876 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:41.968935 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:41.969895 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.970509 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.971572 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.971796 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.971861 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:41.974175 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:41.974177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:41.974177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:41.974185 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:41.975912 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:41.976348 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:41.976349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:41.976350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:41.976407 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:41.976683 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:41.976684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:41.976685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:41.976810 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:41.976885 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:41.977160 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 14:24:41.977238 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:24:41.977562 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:41.968112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.3:48247)
0620 14:24:41.968267 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:41.968310 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:41.968784 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:41.968990 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:41.968993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:41.969570 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:41.969766 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:41.969771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.970608 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.970609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:41.971472 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:41.971475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:41.971902 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:41.971903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:41.972025 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:41.972580 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:41.972591 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:41.974035 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:41.976014 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:41.976021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:41.976024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:41.976258 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:41.976481 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:41.976482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:41.976482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:41.976591 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:41.976877 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:41.976884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:41.977028 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 14:24:41.977140 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:24:41.977319 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260620 14:24:41.978106 6947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34461
I20260620 14:24:41.978209 6947 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:41.978446 6947 heartbeater.cc:511] Master 127.0.98.62:34461 requested a full tablet report, sending...
I20260620 14:24:41.978855 6488 ts_manager.cc:194] Registered new tserver with Master: 4b3a80c21b024326846d3d812178d470 (127.0.98.3:37759)
I20260620 14:24:41.979202 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48247
I20260620 14:24:41.987082 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 14:24:42.012962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36783
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34461,127.0.98.61:36783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:42.096494 6952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:42.096726 6952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:42.096750 6952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:42.097891 6952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:42.097932 6952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:42.097947 6952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:42.097959 6952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:42.097972 6952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:42.099452 6952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34461,127.0.98.61:36783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
W20260620 14:24:42.099556 6952 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:34461,127.0.98.61:36783'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 14:24:42.099673 6952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:42.099858 6952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:42.104938 6958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.105118 6952 server_base.cc:1034] running on GCE node
W20260620 14:24:42.104954 6960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:42.104993 6957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.105501 6952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:42.105836 6952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:42.107013 6952 hybrid_clock.cc:648] HybridClock initialized: now 1781965482107007 us; error 39 us; skew 500 ppm
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:42.109807 6952 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 14:24:42.110280 6952 webserver.cc:460] Webserver started at http://127.0.98.61:44487/ using document root <none> and password file <none>
I20260620 14:24:42.110554 6952 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:42.110626 6952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:42.110775 6952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:42.111864 6952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data/instance:
uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-hfc3"
server_key: "7db5dee50a3010e1e33f79a3253327f4"
server_key_iv: "1a3f36a618db6f0d7aeb8d304979a80f"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.112231 6952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/wal/instance:
uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-hfc3"
server_key: "7db5dee50a3010e1e33f79a3253327f4"
server_key_iv: "1a3f36a618db6f0d7aeb8d304979a80f"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.113684 6952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:42.114566 6967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.114887 6952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:42.114997 6952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/wal
uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-hfc3"
server_key: "7db5dee50a3010e1e33f79a3253327f4"
server_key_iv: "1a3f36a618db6f0d7aeb8d304979a80f"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.115077 6952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:42.133036 6952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:42.135505 6952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:42.135674 6952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:42.139819 6952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36783
I20260620 14:24:42.139859 7019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36783 every 8 connection(s)
I20260620 14:24:42.140259 6952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:24:42.141273 7020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:42.142877 7020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:24:42.148304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 6952
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:42.153286 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.143671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.0.98.62:34461)
0620 14:24:42.143780 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.143783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.143961 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:24:42.144482 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.144483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.144496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.144580 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.145362 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.145884 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.147189 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.147395 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.147483 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.150046 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.150048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.150048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.150055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.151821 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.152233 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.152235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.152235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.152303 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.152584 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.152585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.152585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.152752 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.152827 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.153105 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.153193 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:24:42.153508 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.143569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:37670)
0620 14:24:42.143823 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.143875 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.144366 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.144663 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.144668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.145029 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.145241 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.145246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.145986 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.145987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.147082 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.147085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.147493 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.147494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.147620 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.148211 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.148220 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.149964 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.151896 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.151901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.151903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.152154 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.152395 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.152396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.152397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.152517 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.152828 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.152835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.152974 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.153080 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.153252 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":103,"threads_started":1}
I20260620 14:24:42.154207 7020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:24:42.158732 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.148842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.0.98.61:36783)
0620 14:24:42.149016 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.149022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.149078 (+ 56us) server_negotiation.cc:407] Connection header received
0620 14:24:42.149108 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.149110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.149131 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.149215 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.149923 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.150828 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.152140 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.152556 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.152639 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.155696 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.155702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.155702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.155708 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.157285 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.157753 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.157757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.157757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.157870 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.158113 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.158114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.158114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.158307 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.158484 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.158601 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.158661 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:24:42.162515 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.154492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36783 (local address 127.0.0.1:33368)
0620 14:24:42.154668 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.154678 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.154724 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.154880 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.154880 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.154987 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.155033 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.155034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.155893 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.155894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.156705 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.156708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.157230 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.157231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.157312 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.157756 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.157761 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.159268 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.161416 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.161417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.161418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.161592 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.161813 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.161813 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.161814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.161868 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.162125 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.162126 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.162162 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.162230 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.162416 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":95}
I20260620 14:24:42.162535 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.154535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.0.98.61:36783)
0620 14:24:42.154695 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.154697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.154714 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:24:42.154781 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.154782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.154794 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.154879 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.155095 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.155821 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.156942 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.157145 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.157212 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.159375 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.159376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.159377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.159383 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.161315 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.161675 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.161676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.161677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.161741 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.161955 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.161956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.161956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.162059 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.162134 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.162256 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.162367 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260620 14:24:42.164016 7020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: Bootstrap starting.
I20260620 14:24:42.164676 7020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:42.164961 7020 log.cc:824] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:42.165733 7020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: No bootstrap required, opened a new log
I20260620 14:24:42.167214 7020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } }
I20260620 14:24:42.167361 7020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:42.167409 7020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2c9d7d05ca4b548a4f9b143e148d1a, State: Initialized, Role: FOLLOWER
I20260620 14:24:42.167557 7020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } }
I20260620 14:24:42.167871 7020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:42.167857 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } } }
I20260620 14:24:42.167997 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:24:42.169682 7038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:42.169744 7038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:42.171553 6952 master_runner.cc:428] Detected that this master 1e2c9d7d05ca4b548a4f9b143e148d1a is joining an existing cluster
I20260620 14:24:42.171594 6952 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:36783
I20260620 14:24:42.179458 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.172640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.0.98.62:34461)
0620 14:24:42.172690 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.172693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.172817 (+ 124us) server_negotiation.cc:407] Connection header received
0620 14:24:42.172890 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.172891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.172903 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.172964 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.173524 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.174258 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.175032 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.175237 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.175323 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.176523 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.176525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.176525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.176532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.178216 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.178655 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.178656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.178657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.178721 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.178895 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.178896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.178897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.179036 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.179113 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.179268 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.179353 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:24:42.179558 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.172645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.0.98.61:36783)
0620 14:24:42.172738 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.172740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.172859 (+ 119us) server_negotiation.cc:407] Connection header received
0620 14:24:42.172934 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.172934 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.172943 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.173006 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.173436 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.174215 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.175348 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.175656 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.175698 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.176785 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.176786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.176787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.176793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.178252 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.178719 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.178719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.178720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.178766 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.178937 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.178938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.178938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.179057 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.179124 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.179319 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.179411 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":74}
I20260620 14:24:42.179504 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.172598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:37674)
0620 14:24:42.172747 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.172758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.172832 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.173013 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.173014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.173316 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.173402 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.173404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.174365 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.174366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.174944 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.174945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.175355 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.175356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.175422 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.175900 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.175905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.176422 (+ 517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.178302 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.178303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.178304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.178580 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.178776 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.178777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.178778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.178840 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.179112 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.179113 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.179158 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.179242 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.179354 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260620 14:24:42.179972 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.172693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36783 (local address 127.0.0.1:33378)
0620 14:24:42.172809 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.172818 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.172878 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.173024 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.173025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.173279 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.173360 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.173362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.174318 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.174319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.175224 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.175226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.175768 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.175769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.175824 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.176231 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.176236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.176682 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.178385 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.178386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.178387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.178645 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.178832 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.178833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.178833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.178888 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.179151 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.179152 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.179196 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.179782 (+ 586us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.179870 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":37,"thread_start_us":76,"threads_started":1}
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:24:42.189702 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.182073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.0.98.61:36783)
0620 14:24:42.182115 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.182117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.182285 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:24:42.182403 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.182404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.182413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.182467 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.183019 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.183486 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.184610 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.184814 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.184870 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.187068 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.187069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.187070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.187077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.188472 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.188887 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.188888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.188889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.188948 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.189147 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.189148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.189149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.189261 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.189338 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.189521 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.189601 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:24:42.189718 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.181988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36783 (local address 127.0.0.1:33390)
0620 14:24:42.182164 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.182216 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.182294 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.182481 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.182484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.182855 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.182921 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.182923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.183566 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.183567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.184533 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.184536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.184890 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.184891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.184995 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.185428 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.185433 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.186978 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.188574 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.188582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.188584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.188799 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.189020 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.189021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.189021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.189086 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.189331 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.189338 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.189394 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.189495 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.189613 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20260620 14:24:42.190286 6488 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:36783: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 36783
}
attrs {
promote: true
}
}
dest_uuid: "f42193629d04488ca85d1942981c8cac"
cas_config_opid_index: -1
I20260620 14:24:42.190673 6488 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } }
I20260620 14:24:42.191154 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1e2c9d7d05ca4b548a4f9b143e148d1a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } }
I20260620 14:24:42.191548 6994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:42.192273 7048 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:36783
I20260620 14:24:42.192502 6952 master.cc:537] Master@127.0.98.61:36783 shutting down...
I20260620 14:24:42.192515 6994 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 FOLLOWER]: Refusing update from remote peer f42193629d04488ca85d1942981c8cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 14:24:42.192821 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } }
I20260620 14:24:42.192904 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:42.192857 7048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:24:42.193610 6475 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 14:24:42.195152 6952 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:42.195227 6952 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:42.195263 6952 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: stopping tablet replica
I20260620 14:24:42.203461 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.203312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36783 (local address 127.0.0.1:33402)
0620 14:24:42.203371 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.203427 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:24:42.203578 6475 consensus_peers.cc:487] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac -> Peer 1e2c9d7d05ca4b548a4f9b143e148d1a (127.0.98.61:36783): Couldn't send request to peer 1e2c9d7d05ca4b548a4f9b143e148d1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:24:42.207986 6952 master.cc:552] Master@127.0.98.61:36783 shutdown complete.
I20260620 14:24:42.208051 6952 master_runner.cc:305] Clearing existing system tablet
I20260620 14:24:42.208189 6952 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:42.208238 6952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:42.209048 6952 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:42.209685 7052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.209887 6952 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:42.209956 6952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:24:42.209987 6952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/wal
uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-hfc3"
server_key: "7db5dee50a3010e1e33f79a3253327f4"
server_key_iv: "1a3f36a618db6f0d7aeb8d304979a80f"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.210733 6952 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:42.212390 6952 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:24:42.212453 6952 log.cc:1196] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 14:24:42.212693 6952 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: Deleting consensus metadata
I20260620 14:24:42.212869 6952 master_runner.cc:315] Copying system tablet from 127.0.98.62:34461
I20260620 14:24:42.213515 6952 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34461
I20260620 14:24:42.220191 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.213823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:37684)
0620 14:24:42.213948 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.213960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.214036 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.214188 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.214189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.214445 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.214511 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.214513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.215206 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.215207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.216106 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.216108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.216468 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.216470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.216536 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.216985 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.216991 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.217393 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.219245 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.219246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.219247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.219420 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.219622 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.219622 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.219623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.219674 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.219935 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.219935 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.219973 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.220035 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.220114 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":84,"threads_started":1}
I20260620 14:24:42.220397 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.213927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.0.98.62:34461)
0620 14:24:42.213982 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.213984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.214017 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:24:42.214101 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.214101 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.214110 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.214169 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.214596 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.215129 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.216194 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.216392 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.216450 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.217481 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.217483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.217483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.217489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.219154 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.219492 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.219494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.219494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.219558 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.219732 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.219733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.219733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.219872 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.219942 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.220085 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.220165 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:24:42.220695 6508 tablet_copy_service.cc:140] P f42193629d04488ca85d1942981c8cac: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1e2c9d7d05ca4b548a4f9b143e148d1a ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37684)
I20260620 14:24:42.220849 6508 tablet_copy_service.cc:161] P f42193629d04488ca85d1942981c8cac: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1e2c9d7d05ca4b548a4f9b143e148d1a at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37684: session id = 1e2c9d7d05ca4b548a4f9b143e148d1a-00000000000000000000000000000000
I20260620 14:24:42.221505 6508 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:24:42.222122 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:24:42.223165 6952 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: tablet copy: Starting download of 0 data blocks...
I20260620 14:24:42.223333 6952 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: tablet copy: Starting download of 1 WAL segments...
I20260620 14:24:42.224282 6952 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:24:42.225076 6508 tablet_copy_service.cc:342] P f42193629d04488ca85d1942981c8cac: Request end of tablet copy session 1e2c9d7d05ca4b548a4f9b143e148d1a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37684
I20260620 14:24:42.225139 6508 tablet_copy_service.cc:433] P f42193629d04488ca85d1942981c8cac: ending tablet copy session 1e2c9d7d05ca4b548a4f9b143e148d1a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1e2c9d7d05ca4b548a4f9b143e148d1a
I20260620 14:24:42.227686 6952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:42.228873 7059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:42.228956 7060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:42.231384 7062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.231643 6952 server_base.cc:1034] running on GCE node
I20260620 14:24:42.231814 6952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:42.231992 6952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:42.233129 6952 hybrid_clock.cc:648] HybridClock initialized: now 1781965482233119 us; error 27 us; skew 500 ppm
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:42.234598 6952 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 14:24:42.234875 6952 webserver.cc:460] Webserver started at http://127.0.98.61:43093/ using document root <none> and password file <none>
I20260620 14:24:42.235000 6952 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:42.235037 6952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:42.235836 6952 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:42.236469 7069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.236608 6952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:42.236701 6952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/wal
uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-hfc3"
server_key: "7db5dee50a3010e1e33f79a3253327f4"
server_key_iv: "1a3f36a618db6f0d7aeb8d304979a80f"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.236753 6952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:42.250104 6952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:42.251951 6952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:42.254837 6952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36783
I20260620 14:24:42.254942 7121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36783 every 8 connection(s)
I20260620 14:24:42.255187 6952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:24:42.255901 7122 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:24:42.256318 7122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: Bootstrap starting.
I20260620 14:24:42.258699 7122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:24:42.258927 7122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: Bootstrap complete.
I20260620 14:24:42.259127 7122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } }
I20260620 14:24:42.259202 7122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e2c9d7d05ca4b548a4f9b143e148d1a, State: Initialized, Role: LEARNER
I20260620 14:24:42.259228 7122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } }
I20260620 14:24:42.259421 7122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:42.259420 7125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } }
I20260620 14:24:42.259531 7125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: This master's current role is: LEARNER
I20260620 14:24:42.261101 7137 catalog_manager.cc:1259] Loaded cluster ID: 0a89a7f453ca47f8afefe52921a46e36
I20260620 14:24:42.261168 7137 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: loading cluster ID for follower catalog manager: success
I20260620 14:24:42.262315 7137 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: acquiring CA information for follower catalog manager: success
I20260620 14:24:42.262684 7137 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:24:42.267896 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.260501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.0.98.61:36783)
0620 14:24:42.260639 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.260641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.260651 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:42.260676 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.260677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.260688 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.260769 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.261444 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.262490 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.263255 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.263577 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.263657 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.265033 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.265035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.265035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.265042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.266784 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.267121 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.267123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.267123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.267187 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.267378 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.267379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.267379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.267514 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.267638 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.267697 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.267794 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 14:24:42.293864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34743
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34461,127.0.98.61:36783,127.0.98.60:34743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:42.383296 7140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:42.383603 7140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:42.383643 7140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:42.384860 7140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:42.384904 7140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:42.384919 7140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:42.384932 7140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:42.384945 7140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:42.386482 7140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34461,127.0.98.61:36783,127.0.98.60:34743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:42.386695 7140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:42.386893 7140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:42.392261 7148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:42.392267 7145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:42.392484 7146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.392666 7140 server_base.cc:1034] running on GCE node
I20260620 14:24:42.392926 7140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:42.393217 7140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:42.394517 7140 hybrid_clock.cc:648] HybridClock initialized: now 1781965482394504 us; error 36 us; skew 500 ppm
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:42.397001 7140 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 14:24:42.397501 7140 webserver.cc:460] Webserver started at http://127.0.98.60:40273/ using document root <none> and password file <none>
I20260620 14:24:42.397748 7140 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:42.397816 7140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:42.397960 7140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:42.399030 7140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data/instance:
uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-hfc3"
server_key: "56777572760061c9634c9f26933a1a41"
server_key_iv: "ac0b2c9e79b47e023f9df3f3fb6a2921"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.399384 7140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/wal/instance:
uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-hfc3"
server_key: "56777572760061c9634c9f26933a1a41"
server_key_iv: "ac0b2c9e79b47e023f9df3f3fb6a2921"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.400797 7140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:24:42.401567 7155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.401755 7140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:42.401855 7140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/wal
uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-hfc3"
server_key: "56777572760061c9634c9f26933a1a41"
server_key_iv: "ac0b2c9e79b47e023f9df3f3fb6a2921"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.401942 7140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:42.412820 7140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:42.414907 7140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:42.415122 7140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:42.421480 7140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34743
I20260620 14:24:42.421649 7207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34743 every 8 connection(s)
I20260620 14:24:42.421993 7140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:24:42.423100 7208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:42.425041 7208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:24:42.430220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 7140
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:42.435428 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.425961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.0.98.62:34461)
0620 14:24:42.426008 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.426011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.426153 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:24:42.426810 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.426811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.426820 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.426898 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.427552 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.428047 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.429161 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.429340 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.429388 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.432265 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.432266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.432266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.432274 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.433909 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.434384 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.434385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.434386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.434452 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.434723 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.434724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.434724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.434880 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.434947 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.435238 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.435320 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:24:42.435648 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.425759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:37700)
0620 14:24:42.425961 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.426003 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.426715 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.426955 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.426959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.427221 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.427477 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.427481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.428131 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.428132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.429081 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.429083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.429410 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.429411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.429538 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.430082 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.430093 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.432174 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.433979 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.433986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.433988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.434272 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.434522 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.434523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.434523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.434653 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.434957 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.434964 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.435094 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.435217 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.435377 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260620 14:24:42.436333 7208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:24:42.440917 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.430849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.0.98.60:34743)
0620 14:24:42.431057 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.431061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.431100 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:24:42.431122 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.431124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.431146 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.431354 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.432161 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.432858 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.433964 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.434450 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.434531 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.437347 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.437355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.437355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.437363 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.439611 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.440032 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.440035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.440035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.440092 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.440291 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.440292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.440292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.440434 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.440643 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.440694 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.440769 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":140,"threads_started":1}
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:24:42.444160 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.436665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.0.98.61:36783)
0620 14:24:42.436735 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.436737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.436778 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:24:42.436910 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.436911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.436921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.436971 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.437341 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.438178 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.439117 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.439441 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.439524 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.441558 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.441560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.441561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.441569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.443026 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.443386 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.443388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.443388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.443446 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.443638 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.443640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.443640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.443751 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.443824 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.443993 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.444074 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:24:42.444156 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.436598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36783 (local address 127.0.0.1:33424)
0620 14:24:42.436733 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.436745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.436800 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.436992 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.436992 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.437139 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.437253 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.437256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.438425 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.438426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.439027 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.439029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.439532 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.439534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.439614 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.439980 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.439985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.441458 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.443125 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.443126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.443126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.443295 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.443512 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.443512 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.443513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.443567 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.443816 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.443817 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.443858 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.443967 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.444063 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 14:24:42.445019 7208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:24:42.451574 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.445327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34743 (local address 127.0.0.1:43740)
0620 14:24:42.445408 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.445419 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.445463 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.445573 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.445573 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.445679 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.445708 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.445709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.446382 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.446383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.446888 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.446890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.447401 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.447402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.447438 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.447845 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.447850 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.449038 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.450552 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.450552 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.450553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.450716 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.450934 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.450934 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.450935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.450992 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.451233 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.451234 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.451270 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.451355 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.451447 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 14:24:42.451644 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.445410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43740 (local address 127.0.98.60:34743)
0620 14:24:42.445464 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.445466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.445473 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:24:42.445518 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.445519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.445527 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.445571 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.445759 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.446273 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.446989 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.447306 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.447400 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.449134 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.449135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.449135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.449142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.450464 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.450813 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.450814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.450814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.450869 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.451054 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.451054 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.451055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.451163 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.451235 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.451379 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.451498 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":39}
I20260620 14:24:42.453476 7208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: Bootstrap starting.
I20260620 14:24:42.454643 7208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:42.455053 7208 log.cc:824] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:42.455839 7208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: No bootstrap required, opened a new log
I20260620 14:24:42.458010 7208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } }
I20260620 14:24:42.458173 7208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:42.458196 7208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f6a88aa3e0d4d33b3d13b28c4bc2910, State: Initialized, Role: FOLLOWER
I20260620 14:24:42.458292 7208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } }
I20260620 14:24:42.459076 7208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:42.459053 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } } }
I20260620 14:24:42.459136 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:24:42.461794 7224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:42.461866 7224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:42.464164 7140 master_runner.cc:428] Detected that this master 3f6a88aa3e0d4d33b3d13b28c4bc2910 is joining an existing cluster
I20260620 14:24:42.464208 7140 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:34743
I20260620 14:24:42.473268 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.465530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.0.98.61:36783)
0620 14:24:42.465580 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.465583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.466422 (+ 839us) server_negotiation.cc:407] Connection header received
0620 14:24:42.466490 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.466491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.466502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.466569 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.466973 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.467755 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.468844 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.469164 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.469205 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.470463 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.470465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.470465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.470473 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.471981 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.472409 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.472410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.472410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.472477 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.472692 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.472693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.472693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.472823 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.472907 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.473087 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.473165 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:24:42.473300 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.465637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36783 (local address 127.0.0.1:33428)
0620 14:24:42.466255 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.466265 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.466433 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.466618 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.466619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.466800 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.466875 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.466877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.467849 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.467849 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.468733 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.468734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.469306 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.469306 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.469373 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.469894 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.469899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.470318 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.472091 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.472093 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.472093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.472304 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.472549 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.472550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.472550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.472614 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.472897 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.472898 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.472957 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.473047 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.473162 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":136,"threads_started":1}
I20260620 14:24:42.490911 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.465537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.0.98.62:34461)
0620 14:24:42.465582 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.465584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.465933 (+ 349us) server_negotiation.cc:407] Connection header received
0620 14:24:42.466019 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.466020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.466032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.466096 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.466672 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.467556 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.468834 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.469154 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.469202 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.487645 (+ 18443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.487647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.487647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.487664 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.489465 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.489954 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.489955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.489956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.490043 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.490249 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.490250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.490251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.490433 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.490480 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.490694 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.490801 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:24:42.491019 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.466148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.0.98.60:34743)
0620 14:24:42.466218 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.466220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.466667 (+ 447us) server_negotiation.cc:407] Connection header received
0620 14:24:42.466749 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.466749 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.466758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.466784 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.467225 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.467926 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.469050 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.469320 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.469358 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.487997 (+ 18639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.487998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.487999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.488006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.489632 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.490061 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.490062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.490062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.490116 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.490370 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.490371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.490371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.490498 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.490585 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.490812 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.490886 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:24:42.491165 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.465799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34743 (local address 127.0.0.1:43742)
0620 14:24:42.466545 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.466554 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.466685 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.466836 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.466837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.467083 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.467139 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.467141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.468129 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.468130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.468959 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.468962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.469425 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.469425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.469484 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.487462 (+ 17978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.487468 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.487890 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.489724 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.489725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.489726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.489984 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.490181 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.490182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.490182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.490291 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.490610 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.490610 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.490687 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.490931 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.491029 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":222,"threads_started":1}
I20260620 14:24:42.490892 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.465456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:37714)
0620 14:24:42.465748 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.465769 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.465949 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.466101 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.466102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.466460 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.466566 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.466568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.467654 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.467655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.468713 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.468715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.469304 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.469304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.469373 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.487110 (+ 17737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.487116 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.487500 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.489571 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.489573 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.489574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.489866 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.490104 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.490105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.490105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.490164 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.490528 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.490529 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.490571 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.490639 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.490753 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":5,"thread_start_us":186,"threads_started":1}
I20260620 14:24:42.492180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 6550
I20260620 14:24:42.498422 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44637
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34461
--webserver_port=37137
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34461,127.0.98.61:36783
--tserver_master_addrs=127.0.98.62:34461,127.0.98.61:36783,127.0.98.60:34743 with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:24:42.506148 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.496105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.0.98.60:34743)
0620 14:24:42.496149 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.496152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.497985 (+ 1833us) server_negotiation.cc:407] Connection header received
0620 14:24:42.498071 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.498072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.498083 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.498147 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.498556 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.499171 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.500197 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.500501 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.500577 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.502866 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.502867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.502868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.502877 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.504803 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.505237 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.505239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.505239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.505318 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.505551 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.505552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.505552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.505709 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.505774 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.505971 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.506050 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:24:42.506613 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.496054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34743 (local address 127.0.0.1:43752)
0620 14:24:42.497918 (+ 1864us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.497929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.498000 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.498152 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.498153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.498382 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.498437 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.498439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.499281 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.499283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.500099 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.500102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.500587 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.500589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.500686 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.501140 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.501147 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.502786 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.504921 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.504922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.504923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.505153 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.505378 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.505379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.505379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.505468 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.505768 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.505770 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.505834 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.505927 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.506064 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1843}
I20260620 14:24:42.507499 6488 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:34743: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 34743
}
attrs {
promote: true
}
}
dest_uuid: "f42193629d04488ca85d1942981c8cac"
cas_config_opid_index: 5
I20260620 14:24:42.507747 6488 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } }
I20260620 14:24:42.508278 7234 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3f6a88aa3e0d4d33b3d13b28c4bc2910 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.508800 7182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:42.509752 7182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 FOLLOWER]: Refusing update from remote peer f42193629d04488ca85d1942981c8cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:24:42.510035 7048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:42.511859 7182 raft_consensus.cc:858] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } }
W20260620 14:24:42.511933 7182 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } }
I20260620 14:24:42.512768 7182 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1e2c9d7d05ca4b548a4f9b143e148d1a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } }
I20260620 14:24:42.513983 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.508371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36783 (local address 127.0.0.1:33436)
0620 14:24:42.510059 (+ 1688us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.510070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.510114 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.510360 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.510361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.510362 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:42.510405 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.510407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.511445 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.511446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.512670 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.512673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.513717 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.513718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.513783 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.513806 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.513856 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.513909 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1671}
I20260620 14:24:42.514248 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.508468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33436 (local address 127.0.98.61:36783)
0620 14:24:42.508510 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.508512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.510170 (+ 1658us) server_negotiation.cc:407] Connection header received
0620 14:24:42.510200 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.510201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.510225 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.510281 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:42.510479 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.511242 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.512850 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.513474 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.513543 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.513615 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.514143 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.514184 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:24:42.514495 7096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 LEARNER]: Refusing update from remote peer f42193629d04488ca85d1942981c8cac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:24:42.514775 7048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 14:24:42.515013 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42193629d04488ca85d1942981c8cac. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } }
I20260620 14:24:42.515093 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:24:42.515129 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } }
I20260620 14:24:42.515169 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:24:42.515364 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } }
I20260620 14:24:42.515420 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:24:42.515343 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3f6a88aa3e0d4d33b3d13b28c4bc2910 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.510030 7234 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:34743
I20260620 14:24:42.515964 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.516059 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:42.516469 7125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42193629d04488ca85d1942981c8cac. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.516558 7125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: This master's current role is: LEARNER
I20260620 14:24:42.516599 7125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.516647 7125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: This master's current role is: LEARNER
I20260620 14:24:42.516789 7140 master.cc:537] Master@127.0.98.60:34743 shutting down...
I20260620 14:24:42.517428 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: true } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.517514 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: This master's current role is: LEADER
W20260620 14:24:42.518556 6473 consensus_peers.cc:487] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac -> Peer 3f6a88aa3e0d4d33b3d13b28c4bc2910 (127.0.98.60:34743): Couldn't send request to peer 3f6a88aa3e0d4d33b3d13b28c4bc2910. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:24:42.519786 7140 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 14:24:42.519901 7140 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 14:24:42.519935 7140 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: stopping tablet replica
I20260620 14:24:42.523128 7048 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac: attempting to promote NON_VOTER 1e2c9d7d05ca4b548a4f9b143e148d1a to VOTER
I20260620 14:24:42.523376 7048 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } }
I20260620 14:24:42.523770 7096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 LEARNER]: Refusing update from remote peer f42193629d04488ca85d1942981c8cac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 14:24:42.523963 7047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 14:24:42.524297 6473 consensus_peers.cc:487] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac -> Peer 3f6a88aa3e0d4d33b3d13b28c4bc2910 (127.0.98.60:34743): Couldn't send request to peer 3f6a88aa3e0d4d33b3d13b28c4bc2910. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:24:42.524526 7047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1e2c9d7d05ca4b548a4f9b143e148d1a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.525599 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.525674 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:42.525724 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.525765 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:42.526193 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1e2c9d7d05ca4b548a4f9b143e148d1a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.527477 7240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.527631 7240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:24:42.540004 7140 master.cc:552] Master@127.0.98.60:34743 shutdown complete.
I20260620 14:24:42.540076 7140 master_runner.cc:305] Clearing existing system tablet
I20260620 14:24:42.540228 7140 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:42.540280 7140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:42.541066 7140 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:24:42.541628 7245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.541777 7140 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:42.541841 7140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:42.541869 7140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/wal
uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-hfc3"
server_key: "56777572760061c9634c9f26933a1a41"
server_key_iv: "ac0b2c9e79b47e023f9df3f3fb6a2921"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.542598 7140 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:42.543965 7140 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:24:42.544034 7140 log.cc:1196] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 14:24:42.544286 7140 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: Deleting consensus metadata
I20260620 14:24:42.544413 7140 master_runner.cc:315] Copying system tablet from 127.0.98.62:34461
I20260620 14:24:42.545075 7140 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34461
I20260620 14:24:42.552747 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.545505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.0.98.62:34461)
0620 14:24:42.545554 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.545556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.545658 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:24:42.545711 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.545712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.545724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.545805 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.546181 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.547006 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.548144 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.548429 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.548507 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.549644 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.549645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.549645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.549653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.551373 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.551801 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.551803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.551803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.551872 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.552100 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.552101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.552102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.552245 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.552368 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.552566 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.552652 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:24:42.552807 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.545451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:37718)
0620 14:24:42.545578 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.545589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.545653 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.545819 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.545820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.546031 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.546084 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.546087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.547088 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.547088 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.548045 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.548048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.548515 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.548516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.548574 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.549135 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.549140 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.549542 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.551465 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.551466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.551467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.551692 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.551957 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.551958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.551958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.552025 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.552368 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.552369 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.552431 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.552521 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.552658 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260620 14:24:42.553646 6508 tablet_copy_service.cc:140] P f42193629d04488ca85d1942981c8cac: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3f6a88aa3e0d4d33b3d13b28c4bc2910 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37718)
I20260620 14:24:42.553718 6508 tablet_copy_service.cc:161] P f42193629d04488ca85d1942981c8cac: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3f6a88aa3e0d4d33b3d13b28c4bc2910 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37718: session id = 3f6a88aa3e0d4d33b3d13b28c4bc2910-00000000000000000000000000000000
I20260620 14:24:42.554370 6508 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:24:42.554864 7140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:24:42.556219 7140 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: tablet copy: Starting download of 0 data blocks...
I20260620 14:24:42.556362 7140 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: tablet copy: Starting download of 1 WAL segments...
I20260620 14:24:42.557219 7140 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:24:42.558137 6508 tablet_copy_service.cc:342] P f42193629d04488ca85d1942981c8cac: Request end of tablet copy session 3f6a88aa3e0d4d33b3d13b28c4bc2910-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37718
I20260620 14:24:42.558202 6508 tablet_copy_service.cc:433] P f42193629d04488ca85d1942981c8cac: ending tablet copy session 3f6a88aa3e0d4d33b3d13b28c4bc2910-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3f6a88aa3e0d4d33b3d13b28c4bc2910
I20260620 14:24:42.561043 7140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:42.562361 7255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:42.562520 7252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.566552 7140 server_base.cc:1034] running on GCE node
W20260620 14:24:42.567431 7253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.567670 7140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:42.567838 7140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:42.568972 7140 hybrid_clock.cc:648] HybridClock initialized: now 1781965482568967 us; error 57 us; skew 500 ppm
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:42.570572 7140 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 14:24:42.570864 7140 webserver.cc:460] Webserver started at http://127.0.98.60:42383/ using document root <none> and password file <none>
I20260620 14:24:42.571002 7140 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:42.571043 7140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:42.571821 7140 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:42.572443 7262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.572613 7140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:42.572674 7140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/wal
uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910"
format_stamp: "Formatted at 2026-06-20 14:24:42 on dist-test-slave-hfc3"
server_key: "56777572760061c9634c9f26933a1a41"
server_key_iv: "ac0b2c9e79b47e023f9df3f3fb6a2921"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.572736 7140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:42.603652 7140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:42.605866 7140 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260620 14:24:42.607389 7233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:42.607666 7233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:42.607709 7233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:24:42.609153 7140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34743
I20260620 14:24:42.609189 7317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34743 every 8 connection(s)
I20260620 14:24:42.609493 7140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
W20260620 14:24:42.609505 7233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:42.609560 7233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:42.609611 7233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:42.610450 7318 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:24:42.611173 7318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: Bootstrap starting.
I20260620 14:24:42.611966 7233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37137
--tserver_master_addrs=127.0.98.62:34461,127.0.98.61:36783,127.0.98.60:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:42.612279 7233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:42.612565 7233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:24:42.613492 7318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:24:42.613726 7318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: Bootstrap complete.
I20260620 14:24:42.613936 7318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } }
I20260620 14:24:42.614024 7318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f6a88aa3e0d4d33b3d13b28c4bc2910, State: Initialized, Role: LEARNER
I20260620 14:24:42.614060 7318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } }
I20260620 14:24:42.614221 7318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:42.614450 7326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.614630 7326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:24:42.616611 7337 catalog_manager.cc:1259] Loaded cluster ID: 0a89a7f453ca47f8afefe52921a46e36
I20260620 14:24:42.616708 7337 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: loading cluster ID for follower catalog manager: success
I20260620 14:24:42.617842 7337 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: acquiring CA information for follower catalog manager: success
I20260620 14:24:42.618254 7337 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:24:42.619599 7322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:42.619599 7323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.619717 7233 server_base.cc:1034] running on GCE node
W20260620 14:24:42.619599 7325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.620023 7233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:42.620337 7233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:42.623592 7233 hybrid_clock.cc:648] HybridClock initialized: now 1781965482623586 us; error 45 us; skew 500 ppm
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:42.626258 7233 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:24:42.626816 7233 webserver.cc:460] Webserver started at http://127.0.98.1:37137/ using document root <none> and password file <none>
I20260620 14:24:42.627079 7233 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:42.627139 7233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:42.628671 7233 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:24:42.629420 7344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.629596 7233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:42.629707 7233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "0ff984641c0d4254878dd7c59a393d5d"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "a5d4bfda6f2caf49449ae418947bfe77"
server_key_iv: "bdb8e92765d010276991fd60ffc24569"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.630067 7233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:42.643453 7233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:42.645989 7233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:42.646164 7233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:42.646473 7233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:42.646857 7233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:42.646893 7233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.646944 7233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:42.646970 7233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.658607 7233 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44637
I20260620 14:24:42.658711 7457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44637 every 8 connection(s)
I20260620 14:24:42.659112 7233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:42.664784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 7233
I20260620 14:24:42.664956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 6684
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:42.672132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35279
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34461
--webserver_port=42641
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34461,127.0.98.61:36783
--tserver_master_addrs=127.0.98.62:34461,127.0.98.61:36783,127.0.98.60:34743 with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:24:42.673951 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.660183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45325 (local address 127.0.98.60:34743)
0620 14:24:42.660881 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.660883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.660896 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:24:42.661601 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.661602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.661617 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.661795 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.663913 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.664558 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.665996 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.666210 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.666296 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.669523 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.669525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.669525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.669533 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.671757 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.672482 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.672484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.672484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.672548 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.672860 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.672861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.672862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.673003 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.673058 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.673729 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.673828 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":56,"threads_started":1}
I20260620 14:24:42.674175 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.660195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34743 (local address 127.0.98.1:45325)
0620 14:24:42.660565 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.660603 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.661490 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.661893 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.661897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.662284 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.662662 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.662668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.664659 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.664660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.665896 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.665899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.666311 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.666311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.666505 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.667286 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.667298 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.669407 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.671861 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.671870 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.671873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.672400 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.672615 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.672616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.672616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.672789 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.673100 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.673113 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.673309 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.673488 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.673903 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":2,"thread_start_us":264,"threads_started":1}
I20260620 14:24:42.674630 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.660875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56353 (local address 127.0.98.62:34461)
0620 14:24:42.660918 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.660920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.661203 (+ 283us) server_negotiation.cc:407] Connection header received
0620 14:24:42.661669 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.661671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.661684 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.661720 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.663369 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.663829 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.666645 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.666916 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.667020 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.670094 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.670095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.670095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.670105 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.671867 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.673377 (+ 1510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.673378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.673379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.673444 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.673708 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.673709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.673709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.673841 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.673886 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.674440 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.674525 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:24:42.674911 7458 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34743
I20260620 14:24:42.675040 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.660755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.1:56353)
0620 14:24:42.660995 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.661011 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.661548 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.661893 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.661898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.662483 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.662662 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.662668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.663908 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.663909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.664955 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.664957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.667040 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.667041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.667149 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.667704 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.667710 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.669993 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.671947 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.671948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.671949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.673282 (+ 1333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.673574 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.673575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.673575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.673631 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.673934 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.673935 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.674270 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.674749 (+ 479us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.674871 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":139,"threads_started":1}
I20260620 14:24:42.675048 7458 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:42.675302 7458 heartbeater.cc:511] Master 127.0.98.60:34743 requested a full tablet report, sending...
I20260620 14:24:42.675375 7459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34461
I20260620 14:24:42.675405 7459 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:42.675457 7459 heartbeater.cc:511] Master 127.0.98.62:34461 requested a full tablet report, sending...
I20260620 14:24:42.675807 6488 ts_manager.cc:194] Re-registered known tserver with Master: 0ff984641c0d4254878dd7c59a393d5d (127.0.98.1:44637)
I20260620 14:24:42.676204 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56353
I20260620 14:24:42.676496 7282 ts_manager.cc:194] Registered new tserver with Master: 0ff984641c0d4254878dd7c59a393d5d (127.0.98.1:44637)
I20260620 14:24:42.677205 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.660195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36783 (local address 127.0.98.1:42757)
0620 14:24:42.661013 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.661022 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.661545 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.661901 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.661902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.662357 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.662662 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.662668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.663410 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.663411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.664681 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.664683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.671622 (+ 6939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.671623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.671717 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.672158 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.672164 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.673764 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.676002 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.676003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.676003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.676225 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.676446 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.676446 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.676447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.676505 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.676799 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.676800 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.676843 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.676915 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.677030 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":24,"thread_start_us":256,"threads_started":1}
I20260620 14:24:42.677539 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.660235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42757 (local address 127.0.98.61:36783)
0620 14:24:42.660282 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.660284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.661229 (+ 945us) server_negotiation.cc:407] Connection header received
0620 14:24:42.661676 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.661677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.661690 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.661722 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.662775 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.663279 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.664782 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.665092 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.665130 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.673877 (+ 8747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.673879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.673879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.673889 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.675918 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.676316 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.676317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.676318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.676377 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.676569 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.676569 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.676570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.676716 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.676773 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.677337 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.677424 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:24:42.678071 7460 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36783
I20260620 14:24:42.678143 7460 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:42.678224 7460 heartbeater.cc:511] Master 127.0.98.61:36783 requested a full tablet report, sending...
I20260620 14:24:42.678684 7086 ts_manager.cc:194] Registered new tserver with Master: 0ff984641c0d4254878dd7c59a393d5d (127.0.98.1:44637)
W20260620 14:24:42.759842 7466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:42.760058 7466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:42.760116 7466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:42.761432 7466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:42.761480 7466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:42.761509 7466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:42.763021 7466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42641
--tserver_master_addrs=127.0.98.62:34461,127.0.98.61:36783,127.0.98.60:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:42.763224 7466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:42.763415 7466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:42.768425 7474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:42.768406 7472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:42.768442 7471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.768636 7466 server_base.cc:1034] running on GCE node
I20260620 14:24:42.768870 7466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:42.769125 7466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:42.770287 7466 hybrid_clock.cc:648] HybridClock initialized: now 1781965482770283 us; error 37 us; skew 500 ppm
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:42.772890 7466 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:24:42.773368 7466 webserver.cc:460] Webserver started at http://127.0.98.2:42641/ using document root <none> and password file <none>
I20260620 14:24:42.773562 7466 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:42.773613 7466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:42.775087 7466 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:42.775637 7481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.775833 7466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:42.775909 7466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "44cafb4598cc46bb94b5495eef807646"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "7317697df9626e6800444d5823bb625c"
server_key_iv: "4fa582a4a39191e4e7a30d4acd9dada8"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.776258 7466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:42.793926 7466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:42.796073 7466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:42.796223 7466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:42.796509 7466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:42.796838 7466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:42.796893 7466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.796928 7466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:42.796957 7466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.803061 7466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35279
I20260620 14:24:42.803107 7594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35279 every 8 connection(s)
I20260620 14:24:42.803476 7466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:42.809808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 7466
I20260620 14:24:42.809978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 6818
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:24:42.815757 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.804832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46051 (local address 127.0.98.61:36783)
0620 14:24:42.804890 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.804892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.805005 (+ 113us) server_negotiation.cc:407] Connection header received
0620 14:24:42.805615 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.805617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.805631 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.805685 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.806657 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.807381 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.808681 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.808889 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.808972 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.811998 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.811999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.812000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.812009 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.813880 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.814511 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.814512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.814513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.814571 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.814941 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.814942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.814943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.815089 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.815149 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.815531 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.815622 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:24:42.816680 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.804651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36783 (local address 127.0.98.2:46051)
0620 14:24:42.804951 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.804961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.805511 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.806246 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.806247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.806460 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.806581 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.806583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.807472 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.807472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.808595 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.808597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.808978 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.808979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.809126 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.809679 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.809689 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.811833 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.813987 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.813994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.813997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.814395 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.814690 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.814690 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.814691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.814842 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.815184 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.815190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.815374 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.816126 (+ 752us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.816331 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":49,"thread_start_us":117,"threads_started":1}
I20260620 14:24:42.816754 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.804965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58479 (local address 127.0.98.62:34461)
0620 14:24:42.805038 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.805047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.805170 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:24:42.805686 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.805688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.805701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.805756 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.806650 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.807139 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.808697 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.809000 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.809040 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.813631 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.813633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.813633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.813642 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.815407 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.815860 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.815861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.815862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.815928 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.816119 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.816120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.816120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.816242 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.816285 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.816554 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.816633 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:24:42.817121 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.804770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.2:58479)
0620 14:24:42.805086 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.805096 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.805520 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.805795 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.805796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.806205 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.806506 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.806509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.807237 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.807237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.808596 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.808598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.809111 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.809112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.809174 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.811170 (+ 1996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.811175 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.813512 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.815522 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.815523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.815523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.815765 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.816005 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.816005 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.816006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.816057 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.816335 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.816336 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.816384 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.816883 (+ 499us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.816995 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260620 14:24:42.817251 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.804298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40881 (local address 127.0.98.60:34743)
0620 14:24:42.804372 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.804389 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.804914 (+ 525us) server_negotiation.cc:407] Connection header received
0620 14:24:42.805627 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.805629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.805645 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.805677 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.806597 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.807381 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.809425 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.809749 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.809810 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.813583 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.813584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.813585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.813593 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.815925 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.816323 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.816325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.816325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.816382 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.816593 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.816594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.816594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.816736 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.816780 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.817047 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.817152 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:24:42.817651 7597 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36783
I20260620 14:24:42.817664 7596 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34461
I20260620 14:24:42.817835 7596 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:42.817834 7597 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:42.818048 7596 heartbeater.cc:511] Master 127.0.98.62:34461 requested a full tablet report, sending...
I20260620 14:24:42.818053 7597 heartbeater.cc:511] Master 127.0.98.61:36783 requested a full tablet report, sending...
I20260620 14:24:42.817687 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.804290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34743 (local address 127.0.98.2:40881)
0620 14:24:42.804761 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.804810 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.805503 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.805780 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.805784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.806186 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.806499 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.806504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.808452 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.808453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.809339 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.809342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.809866 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.809867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.809947 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.811848 (+ 1901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.811854 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.813478 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.816025 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.816026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.816026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.816243 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.816464 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.816465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.816465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.816528 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.816828 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.816829 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.816925 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.817363 (+ 438us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.817480 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":245,"threads_started":1}
I20260620 14:24:42.818240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37759
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34461
--webserver_port=37933
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34461,127.0.98.61:36783
--tserver_master_addrs=127.0.98.62:34461,127.0.98.61:36783,127.0.98.60:34743 with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:24:42.818681 7595 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34743
I20260620 14:24:42.818737 7595 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:42.818809 7595 heartbeater.cc:511] Master 127.0.98.60:34743 requested a full tablet report, sending...
I20260620 14:24:42.818804 7086 ts_manager.cc:194] Registered new tserver with Master: 44cafb4598cc46bb94b5495eef807646 (127.0.98.2:35279)
I20260620 14:24:42.819068 7282 ts_manager.cc:194] Registered new tserver with Master: 44cafb4598cc46bb94b5495eef807646 (127.0.98.2:35279)
I20260620 14:24:42.819707 6488 ts_manager.cc:194] Re-registered known tserver with Master: 44cafb4598cc46bb94b5495eef807646 (127.0.98.2:35279)
I20260620 14:24:42.820053 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58479
W20260620 14:24:42.902011 7602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:42.902231 7602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:42.902256 7602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:42.903474 7602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:42.903518 7602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:42.903545 7602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:42.904953 7602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37933
--tserver_master_addrs=127.0.98.62:34461,127.0.98.61:36783,127.0.98.60:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:42.905140 7602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:42.905320 7602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:42.910336 7608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:42.910356 7610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.910537 7602 server_base.cc:1034] running on GCE node
W20260620 14:24:42.910437 7607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:42.910773 7602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:42.911026 7602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:42.912212 7602 hybrid_clock.cc:648] HybridClock initialized: now 1781965482912202 us; error 29 us; skew 500 ppm
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:42.915400 7602 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:24:42.915969 7602 webserver.cc:460] Webserver started at http://127.0.98.3:37933/ using document root <none> and password file <none>
I20260620 14:24:42.916206 7602 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:42.916306 7602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:42.917670 7602 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:42.918308 7617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.918503 7602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:42.918602 7602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "4b3a80c21b024326846d3d812178d470"
format_stamp: "Formatted at 2026-06-20 14:24:41 on dist-test-slave-hfc3"
server_key: "a9257d2fd81772f699b4cf43c049a3f1"
server_key_iv: "782c72ee42ed166bd1dec58f37434d62"
server_key_version: "encryptionkey@0"
I20260620 14:24:42.919001 7602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:42.943450 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.940475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34743 (local address 127.0.0.1:43766)
0620 14:24:42.940542 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.940561 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.940637 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.940879 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.940880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.940882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:42.940937 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.940939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.941660 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.941661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.942533 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.942536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.943173 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.943174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.943247 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.943251 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.943308 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.943356 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:24:42.943447 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.940554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.0.98.60:34743)
0620 14:24:42.940630 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.940633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.940644 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:24:42.940754 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.940755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.940778 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.940872 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:42.941020 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.941559 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.942636 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.943074 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.943172 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.943269 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.943315 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.943356 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:24:42.943842 7292 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 14:24:42.943991 7326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42193629d04488ca85d1942981c8cac. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.944144 7326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:24:42.958564 7602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:42.961362 7602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:42.961542 7602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:42.961884 7602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:42.962252 7602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:42.962289 7602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.962312 7602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:42.962385 7602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:42.968195 7602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37759
I20260620 14:24:42.968235 7730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37759 every 8 connection(s)
I20260620 14:24:42.968619 7602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:42.974267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 7602
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:24:42 dist-test-slave-hfc3 krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:42.981626 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.969706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34743 (local address 127.0.98.3:49165)
0620 14:24:42.969953 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.969961 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.970871 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.971155 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.971156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.971602 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.971798 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.971804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.972809 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.972811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.974039 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.974040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.974592 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.974593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.974692 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.975233 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.975238 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.977169 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.979502 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.979509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.979512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.980202 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.980441 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.980442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.980442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.980576 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.980912 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.980918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.981085 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.981190 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.981373 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":7,"thread_start_us":65,"threads_started":1}
I20260620 14:24:42.982291 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.969845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49165 (local address 127.0.98.60:34743)
0620 14:24:42.969886 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.969888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.970016 (+ 128us) server_negotiation.cc:407] Connection header received
0620 14:24:42.971023 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.971025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.971041 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.971072 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.971913 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.972693 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.974132 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.974448 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.974491 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.977267 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.977269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.977269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.977276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.979334 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.980293 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.980295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.980295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.980358 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.980650 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.980651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.980652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.980826 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.980885 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.982120 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.982203 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:24:42.982614 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.969470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38711 (local address 127.0.98.62:34461)
0620 14:24:42.969529 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.969532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.969884 (+ 352us) server_negotiation.cc:407] Connection header received
0620 14:24:42.971127 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.971128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.971162 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.971217 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.971913 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.972693 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.973840 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.974150 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.974215 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.978526 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.978528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.978528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.978535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.980976 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.981649 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.981650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.981650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.981712 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.981894 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.981895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.981895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.982010 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.982064 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.982425 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.982505 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:24:42.983031 7731 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34743
I20260620 14:24:42.983148 7731 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:42.983381 7731 heartbeater.cc:511] Master 127.0.98.60:34743 requested a full tablet report, sending...
I20260620 14:24:42.984853 7282 ts_manager.cc:194] Registered new tserver with Master: 4b3a80c21b024326846d3d812178d470 (127.0.98.3:37759)
I20260620 14:24:42.985637 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.969403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.3:38711)
0620 14:24:42.969828 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.969837 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.970860 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.971267 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.971268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.971593 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.971798 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.971804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.972792 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.972793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.973754 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.973757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.974276 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.974277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.974716 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.975356 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.975362 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.978420 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.981348 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.981349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.981349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.981562 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.981776 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.981777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.981777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.981834 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.982087 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.982088 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.982130 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.982198 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.985273 (+ 3075us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":7,"thread_start_us":98,"threads_started":1}
I20260620 14:24:42.986109 7732 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34461
I20260620 14:24:42.986158 7732 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:42.986246 7732 heartbeater.cc:511] Master 127.0.98.62:34461 requested a full tablet report, sending...
I20260620 14:24:42.986583 6488 ts_manager.cc:194] Re-registered known tserver with Master: 4b3a80c21b024326846d3d812178d470 (127.0.98.3:37759)
I20260620 14:24:42.987109 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38711
I20260620 14:24:42.987427 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.969409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54219 (local address 127.0.98.61:36783)
0620 14:24:42.969533 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.969535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.969790 (+ 255us) server_negotiation.cc:407] Connection header received
0620 14:24:42.970977 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.970979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.970993 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.971062 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.971895 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.972487 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.973616 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.973851 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.973916 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.978371 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.978373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.978373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.978382 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.980729 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.986467 (+ 5738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.986468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.986469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.986530 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.986803 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.986804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.986804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.986961 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.987021 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.987223 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.987317 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 14:24:42.987723 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.976088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.0.98.61:36783)
0620 14:24:42.976300 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.976302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.976309 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:24:42.976330 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.976330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.976341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.976403 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.977146 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.977918 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.979554 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.979915 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.980027 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.983305 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.983306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.983307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.983315 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.985368 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.986046 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.986047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.986048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.986114 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.986517 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.986518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.986518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.986748 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.987281 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.987323 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.987409 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":67,"threads_started":1}
I20260620 14:24:42.988065 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.969403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36783 (local address 127.0.98.3:54219)
0620 14:24:42.969661 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.969705 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.970859 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:42.971118 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:42.971121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:42.971589 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:42.971796 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.971801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.972587 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.972588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.973537 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:42.973540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.973941 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.973942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:42.974094 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.974775 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:42.974786 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:42.978265 (+ 3479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:42.985818 (+ 7553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.985818 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.985819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.986376 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.986609 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:42.986610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:42.986610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:42.986675 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:42.987058 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:42.987059 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:42.987106 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:24:42.987566 (+ 460us) client_negotiation.cc:240] Negotiation successful
0620 14:24:42.987728 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":35,"thread_start_us":94,"threads_started":1}
I20260620 14:24:42.988499 7733 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36783
I20260620 14:24:42.988548 7733 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:42.988623 7733 heartbeater.cc:511] Master 127.0.98.61:36783 requested a full tablet report, sending...
I20260620 14:24:42.988760 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.975728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.0.98.60:34743)
0620 14:24:42.977991 (+ 2263us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.977993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.977999 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:24:42.978018 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.978019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.978031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.978091 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.981279 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.981973 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.983354 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.983651 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.983716 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.985444 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.985445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.985446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.985453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.987249 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.987627 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.987628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.987629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.987688 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.987928 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.987929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.987929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.988082 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.988181 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.988533 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.988619 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2243,"thread_start_us":63,"threads_started":1}
I20260620 14:24:42.988878 7086 ts_manager.cc:194] Registered new tserver with Master: 4b3a80c21b024326846d3d812178d470 (127.0.98.3:37759)
I20260620 14:24:42.990307 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.975524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.0.98.62:34461)
0620 14:24:42.977818 (+ 2294us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.977820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.977830 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:42.977852 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:42.977853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:42.977866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:42.980345 (+ 2479us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:42.980386 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.981159 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.982607 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:42.983421 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:42.983776 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:42.985114 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:42.985116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:42.985116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:42.985124 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:42.989354 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.989650 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.989650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.989651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.989702 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:42.989825 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:42.989825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:42.989825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:42.989910 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:42.989984 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:42.990141 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:24:42.990205 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1951,"thread_start_us":110,"threads_started":1}
I20260620 14:24:42.991758 6488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:42.993266 6488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:42.994004 7234 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: true } } }
I20260620 14:24:42.994134 7234 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:42.995031 7749 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac: attempting to promote NON_VOTER 3f6a88aa3e0d4d33b3d13b28c4bc2910 to VOTER
I20260620 14:24:42.995194 7749 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } }
I20260620 14:24:42.995635 7292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 LEARNER]: Refusing update from remote peer f42193629d04488ca85d1942981c8cac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 14:24:42.995738 7096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 FOLLOWER]: Refusing update from remote peer f42193629d04488ca85d1942981c8cac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 14:24:42.995926 7047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 14:24:42.996024 7048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 14:24:42.996457 7234 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3f6a88aa3e0d4d33b3d13b28c4bc2910 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } } }
I20260620 14:24:42.996593 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3f6a88aa3e0d4d33b3d13b28c4bc2910 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } } }
I20260620 14:24:42.997365 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } } }
I20260620 14:24:42.997453 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:42.997416 7749 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } } }
I20260620 14:24:42.997449 7240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } } }
I20260620 14:24:42.997506 7749 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:42.997514 7240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:24:42.997924 7292 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3f6a88aa3e0d4d33b3d13b28c4bc2910 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } } }
I20260620 14:24:42.999114 7326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } } }
I20260620 14:24:42.999204 7326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:24:42.997414 7234 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f42193629d04488ca85d1942981c8cac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } } }
I20260620 14:24:42.999454 7234 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:43.002971 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.998378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35279 (local address 127.0.0.1:41198)
0620 14:24:42.998410 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.998422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.998461 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:43.000207 (+ 1746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:43.000208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:43.000210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:43.000259 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:43.000261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.001096 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.001098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:43.001972 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:43.001974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.002745 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.002746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:43.002802 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.002805 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:43.002863 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:24:43.002904 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 14:24:43.003186 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.998525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.0.98.2:35279)
0620 14:24:42.999954 (+ 1429us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.999960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.999971 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:24:43.000015 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:43.000017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:43.000035 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:43.000192 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:43.000332 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.001005 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.002059 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.002646 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.002743 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.002828 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:43.002879 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:24:43.003002 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1373,"thread_start_us":80,"threads_started":1}
I20260620 14:24:43.003993 7529 tablet_service.cc:1467] Processing CreateTablet for tablet 243e74254ee14b9788f4f9b99dace083 (DEFAULT_TABLE table=test-table [id=f8287c783cbc4d1c96b6bc12037cf243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:43.004426 7529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 243e74254ee14b9788f4f9b99dace083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:43.004463 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.999155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44637 (local address 127.0.0.1:53448)
0620 14:24:42.999723 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.999731 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.999881 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:43.000286 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:43.000287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:43.000290 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:43.000327 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:43.000329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.001674 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.001675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:43.002854 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:43.002857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.004160 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.004161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:43.004238 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.004243 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:24:43.004293 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:24:43.004345 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":104,"threads_started":1}
I20260620 14:24:43.004748 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.999415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.0.98.1:44637)
0620 14:24:42.999628 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:42.999633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:42.999964 (+ 331us) server_negotiation.cc:407] Connection header received
0620 14:24:43.000031 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:43.000034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:43.000051 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:43.000219 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:43.000384 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.001578 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.002947 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.003926 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.003999 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.004121 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:43.004570 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 14:24:43.004630 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":109,"threads_started":1}
I20260620 14:24:43.005431 7392 tablet_service.cc:1467] Processing CreateTablet for tablet 243e74254ee14b9788f4f9b99dace083 (DEFAULT_TABLE table=test-table [id=f8287c783cbc4d1c96b6bc12037cf243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:43.005815 7392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 243e74254ee14b9788f4f9b99dace083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:43.006053 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.998598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37759 (local address 127.0.0.1:46958)
0620 14:24:42.998763 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:42.998773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:42.998947 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:43.000808 (+ 1861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:43.000809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:43.000811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:43.000854 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:43.000855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.002203 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.002204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:43.003287 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:43.003289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.004287 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.004289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:43.004375 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.004380 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:24:43.005913 (+ 1533us) client_negotiation.cc:240] Negotiation successful
0620 14:24:43.005972 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260620 14:24:43.006584 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:42.998768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.0.98.3:37759)
0620 14:24:43.000520 (+ 1752us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:43.000525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:43.000535 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:43.000586 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:43.000588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:43.000604 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:43.000926 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:43.000984 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.002106 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.003419 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.004208 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.004286 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.004403 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:43.006412 (+ 2009us) server_negotiation.cc:299] Negotiation successful
0620 14:24:43.006476 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1696,"thread_start_us":231,"threads_started":1}
I20260620 14:24:43.009033 7761 tablet_bootstrap.cc:492] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646: Bootstrap starting.
I20260620 14:24:43.009613 7762 tablet_bootstrap.cc:492] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d: Bootstrap starting.
I20260620 14:24:43.010044 7761 tablet_bootstrap.cc:654] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:43.009999 7665 tablet_service.cc:1467] Processing CreateTablet for tablet 243e74254ee14b9788f4f9b99dace083 (DEFAULT_TABLE table=test-table [id=f8287c783cbc4d1c96b6bc12037cf243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:43.010635 7665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 243e74254ee14b9788f4f9b99dace083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:43.010695 7761 log.cc:824] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:43.010821 7762 tablet_bootstrap.cc:654] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:43.011442 7762 log.cc:824] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:43.012002 7761 tablet_bootstrap.cc:492] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646: No bootstrap required, opened a new log
I20260620 14:24:43.012073 7761 ts_tablet_manager.cc:1397] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:24:43.012444 7762 tablet_bootstrap.cc:492] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d: No bootstrap required, opened a new log
I20260620 14:24:43.012535 7762 ts_tablet_manager.cc:1397] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:24:43.014436 7761 raft_consensus.cc:348] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } }
I20260620 14:24:43.014664 7761 raft_consensus.cc:374] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:43.014712 7761 raft_consensus.cc:729] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44cafb4598cc46bb94b5495eef807646, State: Initialized, Role: FOLLOWER
I20260620 14:24:43.014854 7761 consensus_queue.cc:260] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } }
I20260620 14:24:43.015450 7761 ts_tablet_manager.cc:1428] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260620 14:24:43.015542 7596 heartbeater.cc:503] Master 127.0.98.62:34461 was elected leader, sending a full tablet report...
I20260620 14:24:43.017678 7762 raft_consensus.cc:348] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } }
I20260620 14:24:43.017836 7762 raft_consensus.cc:374] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:43.017860 7762 raft_consensus.cc:729] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff984641c0d4254878dd7c59a393d5d, State: Initialized, Role: FOLLOWER
I20260620 14:24:43.017985 7762 consensus_queue.cc:260] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } }
I20260620 14:24:43.018509 7762 ts_tablet_manager.cc:1428] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260620 14:24:43.018550 7459 heartbeater.cc:503] Master 127.0.98.62:34461 was elected leader, sending a full tablet report...
I20260620 14:24:43.018543 7766 tablet_bootstrap.cc:492] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470: Bootstrap starting.
I20260620 14:24:43.019580 7766 tablet_bootstrap.cc:654] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:43.020200 7766 log.cc:824] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:43.021047 7766 tablet_bootstrap.cc:492] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470: No bootstrap required, opened a new log
I20260620 14:24:43.021101 7766 ts_tablet_manager.cc:1397] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:24:43.023959 7766 raft_consensus.cc:348] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } }
I20260620 14:24:43.024096 7766 raft_consensus.cc:374] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:43.024117 7766 raft_consensus.cc:729] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3a80c21b024326846d3d812178d470, State: Initialized, Role: FOLLOWER
I20260620 14:24:43.024230 7766 consensus_queue.cc:260] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } }
I20260620 14:24:43.024518 7766 ts_tablet_manager.cc:1428] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 14:24:43.024610 7732 heartbeater.cc:503] Master 127.0.98.62:34461 was elected leader, sending a full tablet report...
W20260620 14:24:43.054999 7598 tablet.cc:2369] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:24:43.161224 7461 tablet.cc:2369] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:43.215634 7767 raft_consensus.cc:482] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:43.215786 7767 raft_consensus.cc:504] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } }
I20260620 14:24:43.216172 7767 leader_election.cc:288] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44cafb4598cc46bb94b5495eef807646 (127.0.98.2:35279), 4b3a80c21b024326846d3d812178d470 (127.0.98.3:37759)
I20260620 14:24:43.219352 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:43.216448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41945 (local address 127.0.98.2:35279)
0620 14:24:43.216510 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:43.216512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:43.216691 (+ 179us) server_negotiation.cc:407] Connection header received
0620 14:24:43.216723 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:43.216724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:43.216738 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:43.216792 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:43.217038 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.217531 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.218572 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.218938 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.218992 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.219001 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:43.219232 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:24:43.219279 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:24:43.219362 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:43.216372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35279 (local address 127.0.98.1:41945)
0620 14:24:43.216570 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:43.216579 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:43.216646 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:43.216812 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:43.216813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:43.216815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:43.216948 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:43.216950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.217616 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.217617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:43.218495 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:43.218497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.219015 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.219016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:43.219121 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.219125 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:24:43.219216 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:24:43.219258 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":64,"threads_started":1}
W20260620 14:24:43.219321 7734 tablet.cc:2369] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:43.219508 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:43.216372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37759 (local address 127.0.98.1:41485)
0620 14:24:43.216522 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:43.216533 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:43.216713 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:43.216884 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:43.216885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:43.216887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:43.216973 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:43.216974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.217647 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.217648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:43.218501 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:43.218503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.219077 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.219078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:43.219171 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.219176 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:24:43.219342 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 14:24:43.219399 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":48,"thread_start_us":72,"threads_started":1}
I20260620 14:24:43.219902 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:43.216465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41485 (local address 127.0.98.3:37759)
0620 14:24:43.216508 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:43.216511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:43.216696 (+ 185us) server_negotiation.cc:407] Connection header received
0620 14:24:43.216791 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:43.216792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:43.216805 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:43.216863 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:43.217071 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.217543 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.218609 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.218990 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.219061 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.219070 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:43.219269 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 14:24:43.219312 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:24:43.219888 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "243e74254ee14b9788f4f9b99dace083" candidate_uuid: "0ff984641c0d4254878dd7c59a393d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44cafb4598cc46bb94b5495eef807646" is_pre_election: true
I20260620 14:24:43.220070 7549 raft_consensus.cc:2454] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ff984641c0d4254878dd7c59a393d5d in term 0.
I20260620 14:24:43.220216 7685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "243e74254ee14b9788f4f9b99dace083" candidate_uuid: "0ff984641c0d4254878dd7c59a393d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b3a80c21b024326846d3d812178d470" is_pre_election: true
I20260620 14:24:43.220363 7685 raft_consensus.cc:2454] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ff984641c0d4254878dd7c59a393d5d in term 0.
I20260620 14:24:43.220399 7347 leader_election.cc:302] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff984641c0d4254878dd7c59a393d5d, 44cafb4598cc46bb94b5495eef807646; no voters:
I20260620 14:24:43.220546 7767 raft_consensus.cc:2781] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:43.220600 7767 raft_consensus.cc:482] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:43.220618 7767 raft_consensus.cc:3037] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:43.221553 7767 raft_consensus.cc:504] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } }
I20260620 14:24:43.221776 7767 leader_election.cc:288] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [CANDIDATE]: Term 1 election: Requested vote from peers 44cafb4598cc46bb94b5495eef807646 (127.0.98.2:35279), 4b3a80c21b024326846d3d812178d470 (127.0.98.3:37759)
I20260620 14:24:43.221956 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "243e74254ee14b9788f4f9b99dace083" candidate_uuid: "0ff984641c0d4254878dd7c59a393d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44cafb4598cc46bb94b5495eef807646"
I20260620 14:24:43.221990 7685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "243e74254ee14b9788f4f9b99dace083" candidate_uuid: "0ff984641c0d4254878dd7c59a393d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b3a80c21b024326846d3d812178d470"
I20260620 14:24:43.222038 7549 raft_consensus.cc:3037] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:43.222072 7685 raft_consensus.cc:3037] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:43.223074 7685 raft_consensus.cc:2454] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ff984641c0d4254878dd7c59a393d5d in term 1.
I20260620 14:24:43.223304 7345 leader_election.cc:302] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff984641c0d4254878dd7c59a393d5d, 4b3a80c21b024326846d3d812178d470; no voters:
I20260620 14:24:43.223424 7767 raft_consensus.cc:2781] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:43.223479 7549 raft_consensus.cc:2454] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ff984641c0d4254878dd7c59a393d5d in term 1.
I20260620 14:24:43.223588 7767 raft_consensus.cc:686] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 1 LEADER]: Becoming Leader. State: Replica: 0ff984641c0d4254878dd7c59a393d5d, State: Running, Role: LEADER
I20260620 14:24:43.223680 7767 consensus_queue.cc:237] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } }
I20260620 14:24:43.224555 6488 catalog_manager.cc:5495] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ff984641c0d4254878dd7c59a393d5d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0ff984641c0d4254878dd7c59a393d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } health_report { overall_health: HEALTHY } } }
I20260620 14:24:43.285890 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:43.283223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.0.98.1:44637)
0620 14:24:43.283270 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:43.283272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:43.283280 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:43.283323 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:43.283324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:43.283343 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:43.283418 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:43.283572 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.284105 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.284945 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.285156 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.285235 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.285548 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:43.285656 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:43.285756 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:43.285775 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 14:24:43.285796 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:24:43.288004 7549 raft_consensus.cc:1261] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646 [term 1 FOLLOWER]: Refusing update from remote peer 0ff984641c0d4254878dd7c59a393d5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:43.288058 7685 raft_consensus.cc:1261] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470 [term 1 FOLLOWER]: Refusing update from remote peer 0ff984641c0d4254878dd7c59a393d5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:43.288394 7772 consensus_queue.cc:1035] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:43.288535 7767 consensus_queue.cc:1035] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:43.292726 7776 mvcc.cc:205] Tried to move back new op lower bound from 7298930619544559616 to 7298930619284873216. Current Snapshot: MvccSnapshot[applied={T|T < 7298930619544559616}]
I20260620 14:24:43.293627 7777 mvcc.cc:205] Tried to move back new op lower bound from 7298930619544559616 to 7298930619284873216. Current Snapshot: MvccSnapshot[applied={T|T < 7298930619544559616}]
I20260620 14:24:43.295215 6487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:43.295318 6487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:43.296756 6487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f42193629d04488ca85d1942981c8cac: Sending DeleteTablet for 3 replicas of tablet 243e74254ee14b9788f4f9b99dace083
I20260620 14:24:43.297235 7529 tablet_service.cc:1514] Processing DeleteTablet for tablet 243e74254ee14b9788f4f9b99dace083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41198
I20260620 14:24:43.297260 7665 tablet_service.cc:1514] Processing DeleteTablet for tablet 243e74254ee14b9788f4f9b99dace083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46958
I20260620 14:24:43.297556 7392 tablet_service.cc:1514] Processing DeleteTablet for tablet 243e74254ee14b9788f4f9b99dace083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53448
I20260620 14:24:43.297593 7785 tablet_replica.cc:330] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646: stopping tablet replica
I20260620 14:24:43.297714 7785 raft_consensus.cc:2229] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:43.297860 7785 raft_consensus.cc:2258] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:43.298298 7786 tablet_replica.cc:330] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d: stopping tablet replica
I20260620 14:24:43.298442 7786 raft_consensus.cc:2229] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:43.298629 7786 raft_consensus.cc:2258] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:43.298722 7785 ts_tablet_manager.cc:1905] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:43.298821 7784 tablet_replica.cc:330] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470: stopping tablet replica
I20260620 14:24:43.298905 7784 raft_consensus.cc:2229] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:43.298991 7784 raft_consensus.cc:2258] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:43.299371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 6456
I20260620 14:24:43.299396 7786 ts_tablet_manager.cc:1905] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:43.300786 7785 ts_tablet_manager.cc:1918] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:43.300865 7785 log.cc:1196] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/243e74254ee14b9788f4f9b99dace083
I20260620 14:24:43.301206 7785 ts_tablet_manager.cc:1939] T 243e74254ee14b9788f4f9b99dace083 P 44cafb4598cc46bb94b5495eef807646: Deleting consensus metadata
I20260620 14:24:43.301996 7786 ts_tablet_manager.cc:1918] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:43.302075 7786 log.cc:1196] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-0/wal/wals/243e74254ee14b9788f4f9b99dace083
I20260620 14:24:43.302366 7786 ts_tablet_manager.cc:1939] T 243e74254ee14b9788f4f9b99dace083 P 0ff984641c0d4254878dd7c59a393d5d: Deleting consensus metadata
I20260620 14:24:43.302863 7784 ts_tablet_manager.cc:1905] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:43.304701 7784 ts_tablet_manager.cc:1918] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:43.304781 7784 log.cc:1196] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/243e74254ee14b9788f4f9b99dace083
I20260620 14:24:43.305068 7784 ts_tablet_manager.cc:1939] T 243e74254ee14b9788f4f9b99dace083 P 4b3a80c21b024326846d3d812178d470: Deleting consensus metadata
W20260620 14:24:43.306414 7347 connection.cc:535] server connection from 127.0.0.1:53448 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 14:24:43.306432 7621 connection.cc:535] server connection from 127.0.0.1:46958 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 14:24:43.306658 7483 connection.cc:535] server connection from 127.0.0.1:41198 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:24:43.315539 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:43.298543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.0.98.60:34743)
0620 14:24:43.298591 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:43.298594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:43.305583 (+ 6989us) server_negotiation.cc:407] Connection header received
0620 14:24:43.305636 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:43.305637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:43.305655 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:43.305709 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:43.306505 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.307310 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.308503 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.308814 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.308859 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.311063 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:43.311064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:43.311065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:43.311072 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:43.313302 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:43.313732 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:43.313733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:43.313733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:43.313791 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:43.313994 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:43.313995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:43.313995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:43.314117 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:43.314202 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:43.315331 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0620 14:24:43.315440 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:24:43.316936 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:43.308832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.0.98.60:34743)
0620 14:24:43.308872 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:43.308874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:43.308953 (+ 79us) server_negotiation.cc:407] Connection header received
0620 14:24:43.309054 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:43.309056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:43.309067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:43.309101 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:43.310073 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.310839 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.311928 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.312211 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.312251 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.314068 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:43.314069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:43.314070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:43.314076 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:43.315749 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:43.316126 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:43.316127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:43.316128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:43.316181 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:43.316374 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:43.316375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:43.316375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:43.316493 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:43.316573 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:43.316771 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 14:24:43.316843 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:24:43.317500 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:43.308953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.0.98.61:36783)
0620 14:24:43.308996 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:43.308997 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:43.309250 (+ 253us) server_negotiation.cc:407] Connection header received
0620 14:24:43.309394 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:43.309395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:43.309412 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:43.309443 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:43.310157 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.310921 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.311940 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:43.312243 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:43.312281 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:43.314029 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:43.314031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:43.314031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:43.314039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:43.315987 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:43.316354 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:43.316355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:43.316356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:43.316413 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:43.316625 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:43.316627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:43.316627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:43.316763 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:43.316837 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:43.317305 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 14:24:43.317410 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:24:44.294068 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.293764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.1:36923)
0620 14:24:44.293916 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.293982 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260620 14:24:44.294068 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.293767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.2:39143)
0620 14:24:44.293930 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.294017 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
W20260620 14:24:44.294479 7459 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34461 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
W20260620 14:24:44.294538 7596 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34461 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
I20260620 14:24:44.295049 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.294749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.3:37773)
0620 14:24:44.294854 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.295002 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":50,"threads_started":1}
W20260620 14:24:44.295269 7732 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34461 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
I20260620 14:24:44.797187 7797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f42193629d04488ca85d1942981c8cac)
I20260620 14:24:44.797308 7797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } }
I20260620 14:24:44.797515 7798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f42193629d04488ca85d1942981c8cac)
I20260620 14:24:44.797631 7797 leader_election.cc:288] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f42193629d04488ca85d1942981c8cac (127.0.98.62:34461), 3f6a88aa3e0d4d33b3d13b28c4bc2910 (127.0.98.60:34743)
I20260620 14:24:44.797639 7798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } }
I20260620 14:24:44.797901 7798 leader_election.cc:288] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f42193629d04488ca85d1942981c8cac (127.0.98.62:34461), 1e2c9d7d05ca4b548a4f9b143e148d1a (127.0.98.61:36783)
I20260620 14:24:44.798136 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.797772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:38196)
0620 14:24:44.798025 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.798092 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":38,"threads_started":1}
I20260620 14:24:44.798305 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.797998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:38210)
0620 14:24:44.798210 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.798262 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
W20260620 14:24:44.798439 7264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
W20260620 14:24:44.798314 7071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
I20260620 14:24:44.798839 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.798762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:38222)
0620 14:24:44.798794 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.798817 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 14:24:44.798902 7264 leader_election.cc:334] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f42193629d04488ca85d1942981c8cac (127.0.98.62:34461): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
I20260620 14:24:44.799209 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.799089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:38234)
0620 14:24:44.799146 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.799183 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:24:44.799300 7071 leader_election.cc:334] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f42193629d04488ca85d1942981c8cac (127.0.98.62:34461): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
I20260620 14:24:44.801481 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.797909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34743 (local address 127.0.0.1:43812)
0620 14:24:44.798046 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.798057 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:44.798107 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:44.798443 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:44.798444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:44.798447 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:44.798504 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.798507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.799434 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.799435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.800596 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.800599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.801271 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.801272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.801327 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.801330 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:44.801375 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:24:44.801408 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":2,"thread_start_us":47,"threads_started":1}
I20260620 14:24:44.801514 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.798196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.0.98.60:34743)
0620 14:24:44.798273 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:44.798275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:44.798282 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:24:44.798306 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:44.798307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:44.798317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:44.798409 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:44.798607 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.799358 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.800681 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.801193 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.801263 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.801275 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:44.801398 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 14:24:44.801442 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60,"thread_start_us":31,"threads_started":1}
I20260620 14:24:44.801697 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" is_pre_election: true
I20260620 14:24:44.801811 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.798452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.0.98.61:36783)
0620 14:24:44.798649 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:44.798651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:44.798659 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:44.798682 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:44.798683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:44.798695 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:44.798773 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:44.798946 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.799603 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.800859 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.801416 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.801479 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.801490 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:44.801699 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 14:24:44.801739 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":176,"thread_start_us":54,"threads_started":1}
I20260620 14:24:44.801813 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.798113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36783 (local address 127.0.0.1:33458)
0620 14:24:44.798242 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.798252 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:44.798310 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:44.798819 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:44.798820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:44.798823 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:44.798866 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.798869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.799681 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.799682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.800784 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.800787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.801495 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.801496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.801591 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.801595 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:24:44.801647 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:24:44.801689 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
I20260620 14:24:44.801820 7292 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2c9d7d05ca4b548a4f9b143e148d1a in term 1.
I20260620 14:24:44.801988 7071 leader_election.cc:302] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2c9d7d05ca4b548a4f9b143e148d1a, 3f6a88aa3e0d4d33b3d13b28c4bc2910; no voters: f42193629d04488ca85d1942981c8cac
I20260620 14:24:44.802107 7797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:24:44.802027 7096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" is_pre_election: true
I20260620 14:24:44.802161 7797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader f42193629d04488ca85d1942981c8cac)
I20260620 14:24:44.802186 7797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:24:44.802933 7797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } }
I20260620 14:24:44.803087 7797 leader_election.cc:288] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [CANDIDATE]: Term 2 election: Requested vote from peers f42193629d04488ca85d1942981c8cac (127.0.98.62:34461), 3f6a88aa3e0d4d33b3d13b28c4bc2910 (127.0.98.60:34743)
I20260620 14:24:44.803098 7096 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f6a88aa3e0d4d33b3d13b28c4bc2910 in current term 2: Already voted for candidate 1e2c9d7d05ca4b548a4f9b143e148d1a in this term.
I20260620 14:24:44.803383 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.803264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:38250)
0620 14:24:44.803333 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.803365 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 14:24:44.803375 7266 leader_election.cc:302] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f6a88aa3e0d4d33b3d13b28c4bc2910; no voters: 1e2c9d7d05ca4b548a4f9b143e148d1a, f42193629d04488ca85d1942981c8cac
I20260620 14:24:44.803556 7798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:24:44.803558 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910"
I20260620 14:24:44.803823 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.803744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:38262)
0620 14:24:44.803775 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.803801 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 14:24:44.803912 7071 leader_election.cc:334] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f42193629d04488ca85d1942981c8cac (127.0.98.62:34461): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
I20260620 14:24:44.804381 7798 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:24:44.804948 7292 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2c9d7d05ca4b548a4f9b143e148d1a in term 2.
I20260620 14:24:44.805125 7071 leader_election.cc:302] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2c9d7d05ca4b548a4f9b143e148d1a, 3f6a88aa3e0d4d33b3d13b28c4bc2910; no voters: f42193629d04488ca85d1942981c8cac
I20260620 14:24:44.805220 7797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:24:44.805348 7797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [term 2 LEADER]: Becoming Leader. State: Replica: 1e2c9d7d05ca4b548a4f9b143e148d1a, State: Running, Role: LEADER
I20260620 14:24:44.805428 7797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } }
I20260620 14:24:44.805765 7807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e2c9d7d05ca4b548a4f9b143e148d1a. Latest consensus state: current_term: 2 leader_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } } }
I20260620 14:24:44.805862 7807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:44.806038 7809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:44.806552 7809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:44.806629 7809 catalog_manager.cc:1259] Loaded cluster ID: 0a89a7f453ca47f8afefe52921a46e36
I20260620 14:24:44.806685 7809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:44.806856 7809 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:44.806943 7809 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: Loaded TSK: 0
I20260620 14:24:44.807082 7809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:24:44.852610 7085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:44.854259 7085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:44.855394 7292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [term 2 FOLLOWER]: Refusing update from remote peer 1e2c9d7d05ca4b548a4f9b143e148d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 14:24:44.855436 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.855308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:38270)
0620 14:24:44.855368 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.855405 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:24:44.855631 7797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 14:24:44.855777 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.855680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.0.1:38284)
0620 14:24:44.855727 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.855756 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:24:44.855878 7071 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a -> Peer f42193629d04488ca85d1942981c8cac (127.0.98.62:34461): Couldn't send request to peer f42193629d04488ca85d1942981c8cac. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:24:44.856343 7798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e2c9d7d05ca4b548a4f9b143e148d1a. Latest consensus state: current_term: 2 leader_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } } }
I20260620 14:24:44.856441 7798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:24:44.856992 7798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } } }
I20260620 14:24:44.857075 7798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6a88aa3e0d4d33b3d13b28c4bc2910 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:24:44.857340 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f42193629d04488ca85d1942981c8cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } peers { permanent_uuid: "1e2c9d7d05ca4b548a4f9b143e148d1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36783 } attrs { promote: false } } peers { permanent_uuid: "3f6a88aa3e0d4d33b3d13b28c4bc2910" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34743 } attrs { promote: false } } }
I20260620 14:24:44.857430 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:44.862493 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.859129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35279 (local address 127.0.0.1:41208)
0620 14:24:44.859187 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.859197 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:44.859235 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:44.859615 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:44.859617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:44.859620 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:44.859679 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.859681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.860690 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.860691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.861453 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.861455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.862208 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.862208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.862265 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.862268 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:44.862339 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:24:44.862406 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 14:24:44.862493 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.859237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.0.98.2:35279)
0620 14:24:44.859466 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:44.859468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:44.859477 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:24:44.859505 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:44.859506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:44.859519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:44.859744 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:44.859783 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.860610 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.861544 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.862126 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.862206 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.862220 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:44.862367 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 14:24:44.862416 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":104,"threads_started":1}
I20260620 14:24:44.862749 7529 tablet_service.cc:1467] Processing CreateTablet for tablet 199df918734a4060916488f75379ea88 (DEFAULT_TABLE table=test-table [id=57ced89b219d47eb83373266267963d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:44.862913 7529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199df918734a4060916488f75379ea88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:44.863168 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.859115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44637 (local address 127.0.0.1:53464)
0620 14:24:44.859153 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.859163 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:44.859205 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:44.859818 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:44.859818 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:44.859820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:44.859847 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.859848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.860759 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.860760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.861457 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.861459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.862967 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.862968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.863028 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.863034 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:24:44.863063 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 14:24:44.863107 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 14:24:44.863437 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.859319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.0.98.1:44637)
0620 14:24:44.859685 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:44.859686 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:44.859692 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:24:44.859741 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:44.859742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:44.859753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:44.859817 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:44.859930 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.860699 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.862254 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.862874 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.863291 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.863308 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:44.863327 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 14:24:44.863369 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":69,"threads_started":1}
I20260620 14:24:44.863626 7392 tablet_service.cc:1467] Processing CreateTablet for tablet 199df918734a4060916488f75379ea88 (DEFAULT_TABLE table=test-table [id=57ced89b219d47eb83373266267963d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:44.863765 7392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199df918734a4060916488f75379ea88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:44.864526 7820 tablet_bootstrap.cc:492] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646: Bootstrap starting.
I20260620 14:24:44.864913 7820 tablet_bootstrap.cc:654] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:44.865025 7821 tablet_bootstrap.cc:492] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d: Bootstrap starting.
I20260620 14:24:44.865430 7821 tablet_bootstrap.cc:654] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:44.865494 7820 tablet_bootstrap.cc:492] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646: No bootstrap required, opened a new log
I20260620 14:24:44.865556 7820 ts_tablet_manager.cc:1397] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:44.865685 7820 raft_consensus.cc:348] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } }
I20260620 14:24:44.865758 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.859335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37759 (local address 127.0.0.1:46974)
0620 14:24:44.859526 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.859552 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:44.859603 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:44.859742 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:44.859743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:44.859745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:44.859800 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.859802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.861576 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.861576 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.863010 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.863012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.865536 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.865537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.865610 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.865616 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:24:44.865649 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 14:24:44.865696 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":65,"threads_started":1}
I20260620 14:24:44.865777 7820 raft_consensus.cc:374] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:44.865808 7820 raft_consensus.cc:729] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44cafb4598cc46bb94b5495eef807646, State: Initialized, Role: FOLLOWER
I20260620 14:24:44.865895 7820 consensus_queue.cc:260] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } }
I20260620 14:24:44.866025 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.859408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.0.98.3:37759)
0620 14:24:44.859545 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:44.859547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:44.859594 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:24:44.859645 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:44.859646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:44.859657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:44.859725 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:44.859889 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.860624 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.863111 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.865416 (+ 2305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.865871 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.865889 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:44.865909 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 14:24:44.865957 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260620 14:24:44.866415 7821 tablet_bootstrap.cc:492] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d: No bootstrap required, opened a new log
I20260620 14:24:44.866403 7665 tablet_service.cc:1467] Processing CreateTablet for tablet 199df918734a4060916488f75379ea88 (DEFAULT_TABLE table=test-table [id=57ced89b219d47eb83373266267963d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:44.866469 7821 ts_tablet_manager.cc:1397] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:44.866524 7665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199df918734a4060916488f75379ea88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:44.866567 7821 raft_consensus.cc:348] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } }
I20260620 14:24:44.866631 7821 raft_consensus.cc:374] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:44.866655 7821 raft_consensus.cc:729] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff984641c0d4254878dd7c59a393d5d, State: Initialized, Role: FOLLOWER
I20260620 14:24:44.866700 7821 consensus_queue.cc:260] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } }
I20260620 14:24:44.866845 7820 ts_tablet_manager.cc:1428] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:44.866863 7821 ts_tablet_manager.cc:1428] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:44.867311 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.867113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.1:58801)
0620 14:24:44.867241 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.867279 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260620 14:24:44.867511 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.866233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.2:50751)
0620 14:24:44.867440 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.867476 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1183,"thread_start_us":74,"threads_started":1}
I20260620 14:24:44.867635 7597 heartbeater.cc:503] Master 127.0.98.61:36783 was elected leader, sending a full tablet report...
I20260620 14:24:44.867871 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.867784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.2:47677)
0620 14:24:44.867818 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.867841 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 14:24:44.868094 7828 tablet_bootstrap.cc:492] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470: Bootstrap starting.
I20260620 14:24:44.868412 7828 tablet_bootstrap.cc:654] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:44.868881 7828 tablet_bootstrap.cc:492] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470: No bootstrap required, opened a new log
I20260620 14:24:44.868922 7828 ts_tablet_manager.cc:1397] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:44.869022 7828 raft_consensus.cc:348] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } }
I20260620 14:24:44.869103 7828 raft_consensus.cc:374] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:44.869117 7828 raft_consensus.cc:729] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3a80c21b024326846d3d812178d470, State: Initialized, Role: FOLLOWER
I20260620 14:24:44.869174 7828 consensus_queue.cc:260] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } }
I20260620 14:24:44.869355 7828 ts_tablet_manager.cc:1428] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:44.869714 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.869499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.3:36885)
0620 14:24:44.869636 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.869671 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":101,"threads_started":1}
I20260620 14:24:44.869797 7733 heartbeater.cc:503] Master 127.0.98.61:36783 was elected leader, sending a full tablet report...
I20260620 14:24:44.870036 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.869902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.3:41537)
0620 14:24:44.869985 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.870011 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 14:24:44.975732 7830 raft_consensus.cc:482] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:44.975843 7830 raft_consensus.cc:504] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } }
I20260620 14:24:44.976212 7830 leader_election.cc:288] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44cafb4598cc46bb94b5495eef807646 (127.0.98.2:35279), 0ff984641c0d4254878dd7c59a393d5d (127.0.98.1:44637)
I20260620 14:24:44.979159 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.976442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35827 (local address 127.0.98.2:35279)
0620 14:24:44.976488 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:44.976489 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:44.976499 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:44.976571 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:44.976572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:44.976580 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:44.976638 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:44.976810 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.977287 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.978255 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.978759 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.978854 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.978866 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:44.979027 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:24:44.979064 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:24:44.979162 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.976378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35279 (local address 127.0.98.3:35827)
0620 14:24:44.976431 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.976443 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:44.976497 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:44.976643 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:44.976643 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:44.976646 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:44.976736 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.976738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.977372 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.977373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.978162 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.978164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.978857 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.978858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.978941 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.978944 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:44.979019 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:24:44.979059 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:24:44.979166 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.976466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47943 (local address 127.0.98.1:44637)
0620 14:24:44.976516 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:44.976519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:44.976693 (+ 174us) server_negotiation.cc:407] Connection header received
0620 14:24:44.976762 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:44.976763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:44.976785 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:44.976832 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:44.976981 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.977451 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.978421 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.978794 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.978876 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.978889 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:44.979059 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:24:44.979098 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:24:44.979188 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.976384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44637 (local address 127.0.98.3:47943)
0620 14:24:44.976613 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.976628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:44.976706 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:44.976842 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:44.976843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:44.976844 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:44.976915 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.976917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.977539 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.977540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.978287 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:44.978289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:44.978914 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:44.978915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:44.978980 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:44.978982 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:24:44.979039 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:24:44.979089 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":68,"thread_start_us":111,"threads_started":1}
I20260620 14:24:44.979482 7412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199df918734a4060916488f75379ea88" candidate_uuid: "4b3a80c21b024326846d3d812178d470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff984641c0d4254878dd7c59a393d5d" is_pre_election: true
I20260620 14:24:44.979645 7412 raft_consensus.cc:2454] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3a80c21b024326846d3d812178d470 in term 0.
I20260620 14:24:44.979645 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199df918734a4060916488f75379ea88" candidate_uuid: "4b3a80c21b024326846d3d812178d470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44cafb4598cc46bb94b5495eef807646" is_pre_election: true
I20260620 14:24:44.979755 7549 raft_consensus.cc:2454] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3a80c21b024326846d3d812178d470 in term 0.
I20260620 14:24:44.979949 7619 leader_election.cc:302] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ff984641c0d4254878dd7c59a393d5d, 44cafb4598cc46bb94b5495eef807646, 4b3a80c21b024326846d3d812178d470; no voters:
I20260620 14:24:44.980095 7830 raft_consensus.cc:2781] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:44.980170 7830 raft_consensus.cc:482] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:44.980199 7830 raft_consensus.cc:3037] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:44.980901 7830 raft_consensus.cc:504] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } }
I20260620 14:24:44.981055 7830 leader_election.cc:288] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [CANDIDATE]: Term 1 election: Requested vote from peers 44cafb4598cc46bb94b5495eef807646 (127.0.98.2:35279), 0ff984641c0d4254878dd7c59a393d5d (127.0.98.1:44637)
I20260620 14:24:44.981226 7412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199df918734a4060916488f75379ea88" candidate_uuid: "4b3a80c21b024326846d3d812178d470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff984641c0d4254878dd7c59a393d5d"
I20260620 14:24:44.981287 7412 raft_consensus.cc:3037] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:44.981263 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199df918734a4060916488f75379ea88" candidate_uuid: "4b3a80c21b024326846d3d812178d470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44cafb4598cc46bb94b5495eef807646"
I20260620 14:24:44.981372 7549 raft_consensus.cc:3037] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:44.981958 7412 raft_consensus.cc:2454] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3a80c21b024326846d3d812178d470 in term 1.
I20260620 14:24:44.981972 7549 raft_consensus.cc:2454] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3a80c21b024326846d3d812178d470 in term 1.
I20260620 14:24:44.982170 7620 leader_election.cc:302] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44cafb4598cc46bb94b5495eef807646, 4b3a80c21b024326846d3d812178d470; no voters:
I20260620 14:24:44.982276 7830 raft_consensus.cc:2781] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:44.982461 7830 raft_consensus.cc:686] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 1 LEADER]: Becoming Leader. State: Replica: 4b3a80c21b024326846d3d812178d470, State: Running, Role: LEADER
I20260620 14:24:44.982542 7830 consensus_queue.cc:237] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } }
I20260620 14:24:44.982834 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:44.982729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.3:56395)
0620 14:24:44.982766 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:44.982803 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 14:24:44.983244 7085 catalog_manager.cc:5495] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b3a80c21b024326846d3d812178d470 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4b3a80c21b024326846d3d812178d470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b3a80c21b024326846d3d812178d470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37759 } health_report { overall_health: HEALTHY } } }
I20260620 14:24:45.007738 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:45.005206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.0.98.3:37759)
0620 14:24:45.005272 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:45.005274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:45.005282 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:45.005328 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:45.005329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:45.005348 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:45.005426 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:45.005575 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:45.006052 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:45.006924 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:45.007118 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:45.007226 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:45.007388 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:45.007495 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:45.007580 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:45.007621 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 14:24:45.007644 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:24:45.008886 7549 raft_consensus.cc:1261] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646 [term 1 FOLLOWER]: Refusing update from remote peer 4b3a80c21b024326846d3d812178d470: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:45.008985 7412 raft_consensus.cc:1261] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d [term 1 FOLLOWER]: Refusing update from remote peer 4b3a80c21b024326846d3d812178d470: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:45.009191 7830 consensus_queue.cc:1035] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44cafb4598cc46bb94b5495eef807646" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:45.009399 7833 consensus_queue.cc:1035] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ff984641c0d4254878dd7c59a393d5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:45.009966 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:45.009864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.2:44099)
0620 14:24:45.009897 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:45.009933 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 14:24:45.010082 7460 heartbeater.cc:503] Master 127.0.98.61:36783 was elected leader, sending a full tablet report...
I20260620 14:24:45.010417 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:45.010289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.1:35073)
0620 14:24:45.010348 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:45.010384 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 14:24:45.011165 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:45.011036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.3:53451)
0620 14:24:45.011105 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:45.011132 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:24:45.011955 7839 mvcc.cc:205] Tried to move back new op lower bound from 7298930626594254848 to 7298930626489077760. Current Snapshot: MvccSnapshot[applied={T|T < 7298930626594254848 or (T in {7298930626594254848})}]
I20260620 14:24:45.013208 7084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:45.013316 7084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:45.013414 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:45.013294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.2:56857)
0620 14:24:45.013357 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:45.013383 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:24:45.013810 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:45.013710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.3:38415)
0620 14:24:45.013747 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:45.013780 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 14:24:45.014014 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:45.013894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.1:34527)
0620 14:24:45.013958 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:45.013982 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:24:45.015196 7084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e2c9d7d05ca4b548a4f9b143e148d1a: Sending DeleteTablet for 3 replicas of tablet 199df918734a4060916488f75379ea88
I20260620 14:24:45.015367 7392 tablet_service.cc:1514] Processing DeleteTablet for tablet 199df918734a4060916488f75379ea88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:45 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53464
I20260620 14:24:45.015528 7529 tablet_service.cc:1514] Processing DeleteTablet for tablet 199df918734a4060916488f75379ea88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:45 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41208
I20260620 14:24:45.015700 7846 tablet_replica.cc:330] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646: stopping tablet replica
I20260620 14:24:45.015799 7846 raft_consensus.cc:2229] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:45.015817 7392 tablet_service.cc:1514] Processing DeleteTablet for tablet 199df918734a4060916488f75379ea88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:45 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53464
I20260620 14:24:45.015905 7845 tablet_replica.cc:330] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d: stopping tablet replica
I20260620 14:24:45.015956 7845 raft_consensus.cc:2229] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:45.016004 7845 raft_consensus.cc:2258] T 199df918734a4060916488f75379ea88 P 0ff984641c0d4254878dd7c59a393d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:45.015969 7846 raft_consensus.cc:2258] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:45.016057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 7233
I20260620 14:24:45.016772 7846 ts_tablet_manager.cc:1905] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:45.018059 7846 ts_tablet_manager.cc:1918] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:45.018134 7846 log.cc:1196] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/199df918734a4060916488f75379ea88
I20260620 14:24:45.018435 7846 ts_tablet_manager.cc:1939] T 199df918734a4060916488f75379ea88 P 44cafb4598cc46bb94b5495eef807646: Deleting consensus metadata
I20260620 14:24:45.018761 7072 catalog_manager.cc:4841] TS 44cafb4598cc46bb94b5495eef807646 (127.0.98.2:35279): tablet 199df918734a4060916488f75379ea88 (table test-table [id=57ced89b219d47eb83373266267963d4]) successfully deleted
I20260620 14:24:45.019342 7665 tablet_service.cc:1514] Processing DeleteTablet for tablet 199df918734a4060916488f75379ea88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:45 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46974
I20260620 14:24:45.019769 7847 tablet_replica.cc:330] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470: stopping tablet replica
I20260620 14:24:45.019837 7847 raft_consensus.cc:2229] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:45.019924 7847 raft_consensus.cc:2258] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 14:24:45.024176 7070 connection.cc:535] server connection from 127.0.98.1:42757 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:24:45.024474 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:45.024366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44637 (local address 127.0.0.1:53474)
0620 14:24:45.024415 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:45.024437 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 14:24:45.024595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 7466
W20260620 14:24:45.024609 7071 catalog_manager.cc:4584] TS 0ff984641c0d4254878dd7c59a393d5d (127.0.98.1:44637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 199df918734a4060916488f75379ea88: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44637: connect: Connection refused (error 111)
I20260620 14:24:45.025390 7847 ts_tablet_manager.cc:1905] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:45.027009 7847 ts_tablet_manager.cc:1918] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:45.027076 7847 log.cc:1196] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/199df918734a4060916488f75379ea88
I20260620 14:24:45.027310 7847 ts_tablet_manager.cc:1939] T 199df918734a4060916488f75379ea88 P 4b3a80c21b024326846d3d812178d470: Deleting consensus metadata
I20260620 14:24:45.027784 7070 catalog_manager.cc:4841] TS 4b3a80c21b024326846d3d812178d470 (127.0.98.3:37759): tablet 199df918734a4060916488f75379ea88 (table test-table [id=57ced89b219d47eb83373266267963d4]) successfully deleted
I20260620 14:24:45.030205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 7602
I20260620 14:24:45.036507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 6952
I20260620 14:24:45.041419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 7140
2026-06-20T14:24:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4940 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:45 dist-test-slave-hfc3 krb5kdc[7850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:45 dist-test-slave-hfc3 krb5kdc[7850](info): set up 2 sockets
Jun 20 14:24:45 dist-test-slave-hfc3 krb5kdc[7850](info): commencing operation
krb5kdc: starting...
W20260620 14:24:46.074993 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 14:24:46.190994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:46.271601 7863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:46.271814 7863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:46.271837 7863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:46.272943 7863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:46.272977 7863 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:24:46.272990 7863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:46.273003 7863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:46.273016 7863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:46.273028 7863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:46.274466 7863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34115
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:46.274664 7863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:46.274842 7863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:46.279616 7869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:46.279633 7871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:46.279633 7868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:46.279979 7863 server_base.cc:1034] running on GCE node
I20260620 14:24:46.280227 7863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:46.280503 7863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:46.281647 7863 hybrid_clock.cc:648] HybridClock initialized: now 1781965486281625 us; error 36 us; skew 500 ppm
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:46.284212 7863 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 14:24:46.284592 7863 webserver.cc:460] Webserver started at http://127.0.98.62:42005/ using document root <none> and password file <none>
I20260620 14:24:46.284793 7863 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:46.284853 7863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:46.284943 7863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:46.286000 7863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "0bc73806f64a47a6b4048908fbc2f61c"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-hfc3"
server_key: "ec4c973808a189d857acdbba26dfeb7b"
server_key_iv: "2caea06c1dd63ebc321514515ab390d6"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.286329 7863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "0bc73806f64a47a6b4048908fbc2f61c"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-hfc3"
server_key: "ec4c973808a189d857acdbba26dfeb7b"
server_key_iv: "2caea06c1dd63ebc321514515ab390d6"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.287675 7863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:46.288362 7878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:46.288514 7863 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:46.288605 7863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "0bc73806f64a47a6b4048908fbc2f61c"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-hfc3"
server_key: "ec4c973808a189d857acdbba26dfeb7b"
server_key_iv: "2caea06c1dd63ebc321514515ab390d6"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.288661 7863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:46.307914 7863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:46.310364 7863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:46.310479 7863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:46.314891 7863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34115
I20260620 14:24:46.314982 7940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34115 every 8 connection(s)
I20260620 14:24:46.315323 7863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:46.315972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 7863
I20260620 14:24:46.316049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:46.316040 7941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:46.316267 392 external_mini_cluster.cc:1428] Setting key c666bd12228ba3f27d86f1900cf5c151
I20260620 14:24:46.319468 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c: Bootstrap starting.
I20260620 14:24:46.320158 7941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:46.320461 7941 log.cc:824] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:46.321172 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c: No bootstrap required, opened a new log
I20260620 14:24:46.323128 7941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc73806f64a47a6b4048908fbc2f61c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } }
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:46.323268 7941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:46.323292 7941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc73806f64a47a6b4048908fbc2f61c, State: Initialized, Role: FOLLOWER
I20260620 14:24:46.323441 7941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc73806f64a47a6b4048908fbc2f61c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } }
I20260620 14:24:46.323513 7941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:46.323535 7941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:46.323601 7941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:46.324137 7941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc73806f64a47a6b4048908fbc2f61c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } }
I20260620 14:24:46.324221 7941 leader_election.cc:302] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bc73806f64a47a6b4048908fbc2f61c; no voters:
I20260620 14:24:46.324419 7941 leader_election.cc:288] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:46.324476 7946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:46.324618 7946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [term 1 LEADER]: Becoming Leader. State: Replica: 0bc73806f64a47a6b4048908fbc2f61c, State: Running, Role: LEADER
I20260620 14:24:46.324713 7946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc73806f64a47a6b4048908fbc2f61c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } }
I20260620 14:24:46.324740 7941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:46.326392 7947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bc73806f64a47a6b4048908fbc2f61c. Latest consensus state: current_term: 1 leader_uuid: "0bc73806f64a47a6b4048908fbc2f61c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc73806f64a47a6b4048908fbc2f61c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } }
I20260620 14:24:46.326488 7947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:46.326844 7948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bc73806f64a47a6b4048908fbc2f61c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc73806f64a47a6b4048908fbc2f61c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } }
I20260620 14:24:46.326933 7948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c [sys.catalog]: This master's current role is: LEADER
W20260620 14:24:46.327147 7960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:46.327209 7960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:46.327265 7961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:46.327436 7961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:46.328176 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.316819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.0.98.62:34115)
0620 14:24:46.316996 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.317005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.317039 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:24:46.317629 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.317638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.317704 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.317870 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:46.318533 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.319926 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.320733 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.321239 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.321367 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.323863 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:46.323867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:46.323867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:46.323877 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:46.326123 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.326601 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.326604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.326604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.326715 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.326945 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.326946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.326946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.327279 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:46.327417 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.327842 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.327973 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260620 14:24:46.328838 7961 catalog_manager.cc:1347] Generated new cluster ID: c2be07fa7fc042f7b2bdf5958d2d4c10
I20260620 14:24:46.328883 7961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:46.334841 7961 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:46.335376 7961 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:46.339334 7961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c: Generated new TSK 0
I20260620 14:24:46.339521 7961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 14:24:46.374060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34115
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:46.461030 7968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:46.461256 7968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:46.461280 7968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:46.462603 7968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:46.462651 7968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:46.462673 7968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:24:46.462692 7968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:46.464246 7968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:46.464505 7968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:46.464766 7968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:46.470486 7973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:46.470494 7976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:46.470494 7974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:46.470940 7968 server_base.cc:1034] running on GCE node
I20260620 14:24:46.471180 7968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:46.471421 7968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:46.472611 7968 hybrid_clock.cc:648] HybridClock initialized: now 1781965486472596 us; error 32 us; skew 500 ppm
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:46.475215 7968 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:24:46.475696 7968 webserver.cc:460] Webserver started at http://127.0.98.1:44467/ using document root <none> and password file <none>
I20260620 14:24:46.475936 7968 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:46.476004 7968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:46.476169 7968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:46.477236 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "a20c036d76804f5785c139c24717477b"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-hfc3"
server_key: "7d4661dd5bf12d2e5489ed325202c6ba"
server_key_iv: "4b0dff171d2f3a38a1f35572165caf42"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.477596 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a20c036d76804f5785c139c24717477b"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-hfc3"
server_key: "7d4661dd5bf12d2e5489ed325202c6ba"
server_key_iv: "4b0dff171d2f3a38a1f35572165caf42"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.479004 7968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:24:46.479818 7983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:46.479988 7968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:46.480087 7968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "a20c036d76804f5785c139c24717477b"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-hfc3"
server_key: "7d4661dd5bf12d2e5489ed325202c6ba"
server_key_iv: "4b0dff171d2f3a38a1f35572165caf42"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.480203 7968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:46.494061 7968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:46.495965 7968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:46.496150 7968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:46.496721 7968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:46.496781 7968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:46.496826 7968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:46.496850 7968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:46.504499 7968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39123
I20260620 14:24:46.504568 8098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39123 every 8 connection(s)
I20260620 14:24:46.504937 7968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:46.506273 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.497094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39377 (local address 127.0.98.62:34115)
0620 14:24:46.497147 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.497149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.497386 (+ 237us) server_negotiation.cc:407] Connection header received
0620 14:24:46.497847 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.497848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.497861 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.497948 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:46.498586 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.499112 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.500254 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.500475 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.500547 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.503242 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:46.503245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:46.503245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:46.503255 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:46.505071 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.505525 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.505527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.505527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.505581 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.505772 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.505773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.505773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.505872 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:46.505915 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.506140 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.506209 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:24:46.506531 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.497086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34115 (local address 127.0.98.1:39377)
0620 14:24:46.497243 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:46.497283 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:46.497748 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:46.497998 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:46.498001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:46.498246 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:46.498491 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.498497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.499208 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.499210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.500155 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.500157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.500554 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.500555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.500683 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.501208 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:46.501216 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:46.503127 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:46.505176 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:46.505182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:46.505185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:46.505464 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:46.505640 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:46.505640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:46.505640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:46.505734 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:46.505919 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:46.505922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:46.506033 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 14:24:46.506121 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:24:46.506244 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260620 14:24:46.507058 8099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34115
I20260620 14:24:46.507201 8099 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:46.507396 8099 heartbeater.cc:511] Master 127.0.98.62:34115 requested a full tablet report, sending...
I20260620 14:24:46.507925 7894 ts_manager.cc:194] Registered new tserver with Master: a20c036d76804f5785c139c24717477b (127.0.98.1:39123)
I20260620 14:24:46.508846 7894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39377
I20260620 14:24:46.509991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 7968
I20260620 14:24:46.510104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:46.510311 392 external_mini_cluster.cc:1428] Setting key 576c4bf771db07047ea3c7187828ec90
I20260620 14:24:46.514137 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.508118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35171 (local address 127.0.98.62:34115)
0620 14:24:46.508155 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.508157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.508335 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:24:46.508384 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.508385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.508393 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.508437 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:46.508763 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.509433 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.510408 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.510668 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.510734 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.511727 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:46.511729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:46.511729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:46.511735 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:46.513052 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.513388 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.513389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.513389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.513440 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.513652 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.513653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.513654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.513788 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:46.513849 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.514011 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.514078 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:24:46.514269 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.508078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34115 (local address 127.0.98.1:35171)
0620 14:24:46.508254 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:46.508266 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:46.508329 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:46.508441 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:46.508442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:46.508648 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:46.508702 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.508704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.509535 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.509536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.510264 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.510266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.510755 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.510756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.510822 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.511327 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:46.511334 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:46.511618 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:46.513139 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:46.513140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:46.513140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:46.513314 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:46.513516 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:46.513517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:46.513517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:46.513574 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:46.513860 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:46.513861 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:46.513910 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 14:24:46.513987 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:24:46.514103 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 14:24:46.535122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34115
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:46.617411 8108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:46.617624 8108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:46.617650 8108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:46.618886 8108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:46.618932 8108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:46.618953 8108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:24:46.618999 8108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:46.620486 8108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:46.620734 8108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:46.620970 8108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:46.625934 8114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:46.625934 8116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:46.625946 8113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:46.626534 8108 server_base.cc:1034] running on GCE node
I20260620 14:24:46.626757 8108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:46.627003 8108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:46.628165 8108 hybrid_clock.cc:648] HybridClock initialized: now 1781965486628159 us; error 36 us; skew 500 ppm
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:46.630678 8108 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:24:46.631170 8108 webserver.cc:460] Webserver started at http://127.0.98.2:36533/ using document root <none> and password file <none>
I20260620 14:24:46.631418 8108 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:46.631486 8108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:46.631633 8108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:46.632668 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "1e9be9ff83c14404aa332f9362c43686"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-hfc3"
server_key: "e15712e0e0fe91e705e8d5a197a4d397"
server_key_iv: "a16b6e7afd0d634a74ce75ded298d3cf"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.633033 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1e9be9ff83c14404aa332f9362c43686"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-hfc3"
server_key: "e15712e0e0fe91e705e8d5a197a4d397"
server_key_iv: "a16b6e7afd0d634a74ce75ded298d3cf"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.634423 8108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:24:46.635212 8123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:46.635389 8108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:46.635488 8108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "1e9be9ff83c14404aa332f9362c43686"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-hfc3"
server_key: "e15712e0e0fe91e705e8d5a197a4d397"
server_key_iv: "a16b6e7afd0d634a74ce75ded298d3cf"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.635571 8108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:46.649515 8108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:46.651942 8108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:46.652110 8108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:46.652668 8108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:46.652729 8108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:46.652770 8108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:46.652797 8108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:46.660312 8108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34095
I20260620 14:24:46.660691 8108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:46.660676 8238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34095 every 8 connection(s)
I20260620 14:24:46.662866 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.653114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33981 (local address 127.0.98.62:34115)
0620 14:24:46.653184 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.653187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.653418 (+ 231us) server_negotiation.cc:407] Connection header received
0620 14:24:46.654175 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.654177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.654190 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.654260 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:46.655251 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.655754 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.657457 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.657653 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.657737 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.660083 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:46.660085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:46.660085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:46.660094 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:46.661659 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.662014 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.662015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.662015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.662066 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.662267 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.662267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.662268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.662448 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:46.662502 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.662729 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.662801 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:24:46.663012 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.653052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34115 (local address 127.0.98.2:33981)
0620 14:24:46.653249 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:46.653305 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:46.654040 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:46.654405 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:46.654410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:46.654824 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:46.655134 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.655143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.655835 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.655836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.657365 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.657368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.657756 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.657758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.657894 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.658482 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:46.658491 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:46.659967 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:46.661728 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:46.661732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:46.661734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:46.661947 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:46.662122 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:46.662122 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:46.662123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:46.662215 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:46.662498 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:46.662501 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:46.662616 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 14:24:46.662702 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:24:46.662820 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
I20260620 14:24:46.663475 8239 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34115
I20260620 14:24:46.663548 8239 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:46.663744 8239 heartbeater.cc:511] Master 127.0.98.62:34115 requested a full tablet report, sending...
I20260620 14:24:46.664120 7895 ts_manager.cc:194] Registered new tserver with Master: 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2:34095)
I20260620 14:24:46.664476 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33981
I20260620 14:24:46.669895 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.664593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55301 (local address 127.0.98.62:34115)
0620 14:24:46.664632 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.664634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.664710 (+ 76us) server_negotiation.cc:407] Connection header received
0620 14:24:46.664784 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.664785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.664795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.664838 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:46.665165 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.665628 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.666521 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.666724 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.666813 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.667631 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:46.667632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:46.667633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:46.667640 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:46.668708 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.669081 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.669082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.669082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.669142 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.669340 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.669341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.669342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.669491 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:46.669565 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.669723 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.669805 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:24:46.669934 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.664536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34115 (local address 127.0.98.2:55301)
0620 14:24:46.664651 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:46.664663 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:46.664723 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:46.664852 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:46.664853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:46.665025 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:46.665080 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.665081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.665724 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.665725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.666425 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.666427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.666816 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.666817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.666880 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.667257 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:46.667262 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:46.667542 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:46.668823 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:46.668824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:46.668825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:46.668995 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:46.669207 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:46.669207 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:46.669208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:46.669274 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:46.669569 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:46.669570 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:46.669613 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:24:46.669698 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:24:46.669802 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260620 14:24:46.670161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 8108
I20260620 14:24:46.670282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:46.670532 392 external_mini_cluster.cc:1428] Setting key cb7d38cacad4bbcd2fc2ff8bbd8ef9bd
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:24:46.695485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34115
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:46.778601 8248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:46.778815 8248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:46.778841 8248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:46.780010 8248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:46.780052 8248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:46.780073 8248 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:24:46.780115 8248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:46.781597 8248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:46.781841 8248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:46.782079 8248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:46.786993 8256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:46.787024 8254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:46.787045 8253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:46.787240 8248 server_base.cc:1034] running on GCE node
I20260620 14:24:46.787454 8248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:46.787693 8248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:46.788852 8248 hybrid_clock.cc:648] HybridClock initialized: now 1781965486788854 us; error 45 us; skew 500 ppm
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:46.791497 8248 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:24:46.791882 8248 webserver.cc:460] Webserver started at http://127.0.98.3:33297/ using document root <none> and password file <none>
I20260620 14:24:46.792145 8248 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:46.792192 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:46.792348 8248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:46.793336 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "310d2dad493340c8a942642c21ed2467"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-hfc3"
server_key: "285805d5dd503fd85aeb23fe04de3145"
server_key_iv: "1cdd674b89c09faa863d940721587c53"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.793619 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "310d2dad493340c8a942642c21ed2467"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-hfc3"
server_key: "285805d5dd503fd85aeb23fe04de3145"
server_key_iv: "1cdd674b89c09faa863d940721587c53"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.794916 8248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:46.795571 8263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:46.795759 8248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:46.795847 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "310d2dad493340c8a942642c21ed2467"
format_stamp: "Formatted at 2026-06-20 14:24:46 on dist-test-slave-hfc3"
server_key: "285805d5dd503fd85aeb23fe04de3145"
server_key_iv: "1cdd674b89c09faa863d940721587c53"
server_key_version: "encryptionkey@0"
I20260620 14:24:46.795920 8248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:46.806557 8248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:46.809015 8248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:46.809130 8248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:46.809651 8248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:46.809693 8248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:46.809718 8248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:46.809768 8248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:46.816530 8248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44429
I20260620 14:24:46.816598 8378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44429 every 8 connection(s)
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:46.816978 8248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:46.820029 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.810175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35367 (local address 127.0.98.62:34115)
0620 14:24:46.810239 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.810241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.810522 (+ 281us) server_negotiation.cc:407] Connection header received
0620 14:24:46.810983 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.810985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.810998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.811068 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:46.811913 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.812745 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.814363 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.814670 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.814742 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.817511 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:46.817513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:46.817513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:46.817520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:46.818781 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.819163 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.819163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.819164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.819213 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.819450 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.819450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.819451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.819563 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:46.819615 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.819892 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.819963 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:24:46.820097 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.810171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34115 (local address 127.0.98.3:35367)
0620 14:24:46.810340 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:46.810414 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:46.810887 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:46.811163 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:46.811168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:46.811518 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:46.811809 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.811816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.812869 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.812870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.814243 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.814246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.814751 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.814752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.814889 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.815426 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:46.815437 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:46.817408 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:46.818872 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:46.818878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:46.818879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:46.819094 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:46.819291 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:46.819292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:46.819292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:46.819390 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:46.819618 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:46.819622 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:46.819734 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 14:24:46.819821 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:24:46.819943 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":99,"threads_started":1}
I20260620 14:24:46.820636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 8248
I20260620 14:24:46.820678 8379 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34115
I20260620 14:24:46.820789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:46.820753 8379 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:46.820990 8379 heartbeater.cc:511] Master 127.0.98.62:34115 requested a full tablet report, sending...
I20260620 14:24:46.821237 392 external_mini_cluster.cc:1428] Setting key 02722ffff77a15f270c109d42ef41b6f
I20260620 14:24:46.821475 7894 ts_manager.cc:194] Registered new tserver with Master: 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429)
I20260620 14:24:46.821973 7894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35367
I20260620 14:24:46.822265 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:24:46.828016 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.821841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34115 (local address 127.0.98.3:57035)
0620 14:24:46.821968 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:46.821981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:46.822047 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:46.822214 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:46.822215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:46.822462 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:46.822558 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.822560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.823230 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.823231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.823735 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.823737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.824129 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.824130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.824196 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.824676 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:46.824681 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:46.825043 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:46.826850 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:46.826851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:46.826852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:46.827083 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:46.827351 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:46.827352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:46.827352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:46.827424 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:46.827739 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:46.827740 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:46.827780 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:24:46.827844 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:24:46.827940 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":9,"thread_start_us":73,"threads_started":1}
I20260620 14:24:46.829562 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.821884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57035 (local address 127.0.98.62:34115)
0620 14:24:46.821930 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.821932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.822104 (+ 172us) server_negotiation.cc:407] Connection header received
0620 14:24:46.822126 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.822127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.822138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.822195 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:46.822628 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.823130 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.823840 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.824042 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.824127 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.825135 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:46.825136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:46.825136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:46.825143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:46.826727 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.827180 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.827181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.827182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.827233 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.827493 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.827494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.827495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.827646 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:46.827725 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.829439 (+ 1714us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.829503 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:24:46.844561 7944 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:46.839212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.0.98.62:34115)
0620 14:24:46.839337 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.839339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.839349 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:46.839375 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.839376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.839387 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.839454 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:46.840081 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.840707 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.841374 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.841545 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.841594 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.844455 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51454: BlockingRecv error: recv got EOF from 127.0.0.1:51454 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 14:24:46 dist-test-slave-hfc3 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:46.851565 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.845285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.0.98.62:34115)
0620 14:24:46.845335 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.845337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.845366 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:24:46.845422 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.845423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.845430 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.845479 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:46.845902 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.846412 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.847055 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.847223 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.847274 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.849386 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:46.849387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:46.849387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:46.849393 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:46.850688 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.850980 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.850981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.850981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.851027 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:46.851181 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:46.851182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:46.851182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:46.851275 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:46.851346 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.851448 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.851502 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:24:46.853281 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:46.854894 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:46.861451 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.857154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44429 (local address 127.0.0.1:52884)
0620 14:24:46.857445 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:46.857456 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:46.857506 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:46.858041 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:46.858042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:46.858043 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:46.858100 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.858101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.859407 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.859408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.860366 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.860368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.861202 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.861203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.861266 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.861277 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:24:46.861331 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:24:46.861367 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":97,"threads_started":1}
I20260620 14:24:46.861505 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.857530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.0.98.3:44429)
0620 14:24:46.857871 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.857875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.857885 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:46.857923 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.857925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.857941 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.858191 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:46.858249 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.859348 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.860457 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.861120 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.861199 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.861273 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.861355 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.861394 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":82,"threads_started":1}
I20260620 14:24:46.861698 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.857150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39123 (local address 127.0.0.1:41636)
0620 14:24:46.857325 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:46.857367 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:46.857437 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:46.857655 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:46.857658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:46.857660 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:46.857734 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.857735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.859341 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.859342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.860591 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.860593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.861470 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.861471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.861531 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.861534 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:46.861584 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:24:46.861627 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":52,"thread_start_us":65,"threads_started":1}
I20260620 14:24:46.862131 8313 tablet_service.cc:1467] Processing CreateTablet for tablet ba07b19fa38f40f1b5a92fddcb77ab6c (DEFAULT_TABLE table=test-table [id=663ffc94a56f4f9cbbb5b6ac022dc31a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:46.862582 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba07b19fa38f40f1b5a92fddcb77ab6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:46.862947 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.857388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34095 (local address 127.0.0.1:51320)
0620 14:24:46.857872 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:46.857882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:46.857935 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:46.858182 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:46.858183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:46.858185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:46.858247 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.858249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.859193 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.859194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.860590 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.860593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.862702 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.862703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.862776 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.862781 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:24:46.862843 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:24:46.862882 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":40,"thread_start_us":66,"threads_started":1}
I20260620 14:24:46.862970 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.857282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.0.98.1:39123)
0620 14:24:46.857431 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.857436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.857448 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:24:46.857541 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.857544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.857558 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.857652 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:46.857806 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.859257 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.860679 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.861388 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.861456 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.862779 (+ 1323us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.862831 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.862873 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260620 14:24:46.863000 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.857691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.0.98.2:34095)
0620 14:24:46.857851 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.857855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.857925 (+ 70us) server_negotiation.cc:407] Connection header received
0620 14:24:46.858001 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.858003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.858020 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.858095 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:46.858375 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.859104 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.861880 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.862593 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.862650 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.862738 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.862864 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.862896 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260620 14:24:46.863656 8033 tablet_service.cc:1467] Processing CreateTablet for tablet ba07b19fa38f40f1b5a92fddcb77ab6c (DEFAULT_TABLE table=test-table [id=663ffc94a56f4f9cbbb5b6ac022dc31a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:46.863988 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba07b19fa38f40f1b5a92fddcb77ab6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:46.864037 8173 tablet_service.cc:1467] Processing CreateTablet for tablet ba07b19fa38f40f1b5a92fddcb77ab6c (DEFAULT_TABLE table=test-table [id=663ffc94a56f4f9cbbb5b6ac022dc31a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:46.864343 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba07b19fa38f40f1b5a92fddcb77ab6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:46.865969 8407 tablet_bootstrap.cc:492] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467: Bootstrap starting.
I20260620 14:24:46.866568 8407 tablet_bootstrap.cc:654] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:46.866868 8407 log.cc:824] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:46.867549 8407 tablet_bootstrap.cc:492] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467: No bootstrap required, opened a new log
I20260620 14:24:46.867630 8407 ts_tablet_manager.cc:1397] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:46.867830 8409 tablet_bootstrap.cc:492] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b: Bootstrap starting.
I20260620 14:24:46.868482 8409 tablet_bootstrap.cc:654] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:46.868568 8410 tablet_bootstrap.cc:492] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686: Bootstrap starting.
I20260620 14:24:46.868928 8409 log.cc:824] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:46.869124 8410 tablet_bootstrap.cc:654] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:46.869477 8410 log.cc:824] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:46.869791 8409 tablet_bootstrap.cc:492] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b: No bootstrap required, opened a new log
I20260620 14:24:46.869678 8407 raft_consensus.cc:348] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260620 14:24:46.869835 8407 raft_consensus.cc:374] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:46.869860 8409 ts_tablet_manager.cc:1397] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:46.869870 8407 raft_consensus.cc:729] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310d2dad493340c8a942642c21ed2467, State: Initialized, Role: FOLLOWER
I20260620 14:24:46.869997 8407 consensus_queue.cc:260] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260620 14:24:46.870232 8410 tablet_bootstrap.cc:492] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686: No bootstrap required, opened a new log
I20260620 14:24:46.870285 8379 heartbeater.cc:503] Master 127.0.98.62:34115 was elected leader, sending a full tablet report...
I20260620 14:24:46.870306 8410 ts_tablet_manager.cc:1397] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:46.870559 8407 ts_tablet_manager.cc:1428] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:24:46.871821 8409 raft_consensus.cc:348] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260620 14:24:46.871942 8409 raft_consensus.cc:374] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:46.871964 8409 raft_consensus.cc:729] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20c036d76804f5785c139c24717477b, State: Initialized, Role: FOLLOWER
I20260620 14:24:46.872079 8409 consensus_queue.cc:260] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260620 14:24:46.872212 8410 raft_consensus.cc:348] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260620 14:24:46.872339 8409 ts_tablet_manager.cc:1428] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:24:46.872368 8099 heartbeater.cc:503] Master 127.0.98.62:34115 was elected leader, sending a full tablet report...
I20260620 14:24:46.872366 8410 raft_consensus.cc:374] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:46.872406 8410 raft_consensus.cc:729] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9be9ff83c14404aa332f9362c43686, State: Initialized, Role: FOLLOWER
I20260620 14:24:46.872509 8410 consensus_queue.cc:260] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260620 14:24:46.872759 8410 ts_tablet_manager.cc:1428] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:24:46.872781 8239 heartbeater.cc:503] Master 127.0.98.62:34115 was elected leader, sending a full tablet report...
W20260620 14:24:46.911334 8240 tablet.cc:2369] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:46.981591 8414 raft_consensus.cc:482] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:46.981717 8414 raft_consensus.cc:504] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260620 14:24:46.982126 8414 leader_election.cc:288] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429), 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2:34095)
I20260620 14:24:46.986238 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.982614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56681 (local address 127.0.98.3:44429)
0620 14:24:46.982664 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.982667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.982678 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:24:46.982738 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.982739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.982751 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.982839 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:46.983054 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.983840 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.985331 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.985885 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.985952 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.985961 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.986131 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.986167 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:24:46.986238 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.982806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42375 (local address 127.0.98.2:34095)
0620 14:24:46.982855 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:46.982857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:46.982865 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:46.982947 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:46.982947 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:46.982957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:46.982999 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:46.983153 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.983873 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.985337 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.985874 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.985952 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.985961 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:46.986131 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:24:46.986168 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:24:46.986276 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.982681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34095 (local address 127.0.98.1:42375)
0620 14:24:46.982820 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:46.982830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:46.982874 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:46.983021 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:46.983022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:46.983023 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:46.983088 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.983089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.983938 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.983939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.985232 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.985234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.985963 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.985963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.986046 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.986051 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:24:46.986110 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:24:46.986159 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":60,"thread_start_us":86,"threads_started":1}
I20260620 14:24:46.986286 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:46.982554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44429 (local address 127.0.98.1:56681)
0620 14:24:46.982605 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:46.982618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:46.982668 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:46.982859 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:46.982860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:46.982861 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:46.982956 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.982958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.983926 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.983927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.985231 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:46.985234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:46.985962 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:46.985963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:46.986046 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:46.986051 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:24:46.986110 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:24:46.986159 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":81}
I20260620 14:24:46.986685 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba07b19fa38f40f1b5a92fddcb77ab6c" candidate_uuid: "a20c036d76804f5785c139c24717477b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9be9ff83c14404aa332f9362c43686" is_pre_election: true
I20260620 14:24:46.986822 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba07b19fa38f40f1b5a92fddcb77ab6c" candidate_uuid: "a20c036d76804f5785c139c24717477b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310d2dad493340c8a942642c21ed2467" is_pre_election: true
I20260620 14:24:46.986879 8193 raft_consensus.cc:2454] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20c036d76804f5785c139c24717477b in term 0.
I20260620 14:24:46.986938 8333 raft_consensus.cc:2454] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20c036d76804f5785c139c24717477b in term 0.
I20260620 14:24:46.987138 7985 leader_election.cc:302] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9be9ff83c14404aa332f9362c43686, a20c036d76804f5785c139c24717477b; no voters:
I20260620 14:24:46.987296 8414 raft_consensus.cc:2781] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:46.987396 8414 raft_consensus.cc:482] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:46.987432 8414 raft_consensus.cc:3037] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:46.988139 8414 raft_consensus.cc:504] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260620 14:24:46.988279 8414 leader_election.cc:288] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [CANDIDATE]: Term 1 election: Requested vote from peers 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429), 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2:34095)
I20260620 14:24:46.988421 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba07b19fa38f40f1b5a92fddcb77ab6c" candidate_uuid: "a20c036d76804f5785c139c24717477b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310d2dad493340c8a942642c21ed2467"
I20260620 14:24:46.988474 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba07b19fa38f40f1b5a92fddcb77ab6c" candidate_uuid: "a20c036d76804f5785c139c24717477b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9be9ff83c14404aa332f9362c43686"
I20260620 14:24:46.988519 8333 raft_consensus.cc:3037] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:46.988559 8193 raft_consensus.cc:3037] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:46.989200 8333 raft_consensus.cc:2454] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20c036d76804f5785c139c24717477b in term 1.
I20260620 14:24:46.989265 8193 raft_consensus.cc:2454] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20c036d76804f5785c139c24717477b in term 1.
I20260620 14:24:46.989399 7985 leader_election.cc:302] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 310d2dad493340c8a942642c21ed2467, a20c036d76804f5785c139c24717477b; no voters:
I20260620 14:24:46.989486 8414 raft_consensus.cc:2781] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:46.989670 8414 raft_consensus.cc:686] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 1 LEADER]: Becoming Leader. State: Replica: a20c036d76804f5785c139c24717477b, State: Running, Role: LEADER
I20260620 14:24:46.989778 8414 consensus_queue.cc:237] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260620 14:24:46.990608 7895 catalog_manager.cc:5495] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b reported cstate change: term changed from 0 to 1, leader changed from <none> to a20c036d76804f5785c139c24717477b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a20c036d76804f5785c139c24717477b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } health_report { overall_health: UNKNOWN } } }
W20260620 14:24:47.005973 8100 tablet.cc:2369] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:47.007921 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.005461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.0.98.1:39123)
0620 14:24:47.005527 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.005529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.005538 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:24:47.005630 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.005631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.005649 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.005713 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:47.005902 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.006464 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.007204 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.007381 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.007436 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.007580 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:47.007675 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:47.007774 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.007791 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.007818 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:24:47.009620 8333 raft_consensus.cc:1261] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Refusing update from remote peer a20c036d76804f5785c139c24717477b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:47.009663 8193 raft_consensus.cc:1261] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Refusing update from remote peer a20c036d76804f5785c139c24717477b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:47.010010 8414 consensus_queue.cc:1035] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:47.010174 8417 consensus_queue.cc:1035] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [LEADER]: Connected to new peer: Peer: permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:47.012368 8420 mvcc.cc:205] Tried to move back new op lower bound from 7298930634788675584 to 7298930634710728704. Current Snapshot: MvccSnapshot[applied={T|T < 7298930634788675584}]
I20260620 14:24:47.013036 8422 mvcc.cc:205] Tried to move back new op lower bound from 7298930634788675584 to 7298930634710728704. Current Snapshot: MvccSnapshot[applied={T|T < 7298930634788675584}]
I20260620 14:24:47.013094 8421 mvcc.cc:205] Tried to move back new op lower bound from 7298930634788675584 to 7298930634710728704. Current Snapshot: MvccSnapshot[applied={T|T < 7298930634788675584}]
I20260620 14:24:47.015424 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:47.015522 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:47.016245 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c: Sending DeleteTablet for 3 replicas of tablet ba07b19fa38f40f1b5a92fddcb77ab6c
I20260620 14:24:47.016587 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet ba07b19fa38f40f1b5a92fddcb77ab6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52884
I20260620 14:24:47.016660 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet ba07b19fa38f40f1b5a92fddcb77ab6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51320
I20260620 14:24:47.016754 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet ba07b19fa38f40f1b5a92fddcb77ab6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41636
I20260620 14:24:47.016935 8432 tablet_replica.cc:330] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b: stopping tablet replica
I20260620 14:24:47.017030 8432 raft_consensus.cc:2229] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:47.017086 8431 tablet_replica.cc:330] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686: stopping tablet replica
I20260620 14:24:47.017155 8432 raft_consensus.cc:2258] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:47.017184 8431 raft_consensus.cc:2229] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:47.017207 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:24:47.017328 8431 raft_consensus.cc:2258] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 14:24:47.017338 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:47.017597 8432 ts_tablet_manager.cc:1905] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:47.017771 8431 ts_tablet_manager.cc:1905] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:47.018949 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad6424e365b4263821c7af56edb3523 (DEFAULT_TABLE table=test-table [id=d3e2c66b56194b26abd2a25adee68b87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:47.019048 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad6424e365b4263821c7af56edb3523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.019374 8432 ts_tablet_manager.cc:1918] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:47.019451 8432 log.cc:1196] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/wal/wals/ba07b19fa38f40f1b5a92fddcb77ab6c
I20260620 14:24:47.019501 8431 ts_tablet_manager.cc:1918] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:47.019570 8431 log.cc:1196] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/ba07b19fa38f40f1b5a92fddcb77ab6c
I20260620 14:24:47.019707 8432 ts_tablet_manager.cc:1939] T ba07b19fa38f40f1b5a92fddcb77ab6c P a20c036d76804f5785c139c24717477b: Deleting consensus metadata
I20260620 14:24:47.019855 8431 ts_tablet_manager.cc:1939] T ba07b19fa38f40f1b5a92fddcb77ab6c P 1e9be9ff83c14404aa332f9362c43686: Deleting consensus metadata
I20260620 14:24:47.020218 7880 catalog_manager.cc:4841] TS 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2:34095): tablet ba07b19fa38f40f1b5a92fddcb77ab6c (table test-table [id=663ffc94a56f4f9cbbb5b6ac022dc31a]) successfully deleted
I20260620 14:24:47.020340 8410 tablet_bootstrap.cc:492] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686: Bootstrap starting.
I20260620 14:24:47.020493 7881 catalog_manager.cc:4841] TS a20c036d76804f5785c139c24717477b (127.0.98.1:39123): tablet ba07b19fa38f40f1b5a92fddcb77ab6c (table test-table [id=663ffc94a56f4f9cbbb5b6ac022dc31a]) successfully deleted
I20260620 14:24:47.020524 8313 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad6424e365b4263821c7af56edb3523 (DEFAULT_TABLE table=test-table [id=d3e2c66b56194b26abd2a25adee68b87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:47.020651 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad6424e365b4263821c7af56edb3523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.020640 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad6424e365b4263821c7af56edb3523 (DEFAULT_TABLE table=test-table [id=d3e2c66b56194b26abd2a25adee68b87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:47.020730 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad6424e365b4263821c7af56edb3523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.020748 8410 tablet_bootstrap.cc:654] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.021677 8409 tablet_bootstrap.cc:492] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b: Bootstrap starting.
I20260620 14:24:47.022092 8409 tablet_bootstrap.cc:654] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.022243 8407 tablet_bootstrap.cc:492] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467: Bootstrap starting.
I20260620 14:24:47.022619 8410 tablet_bootstrap.cc:492] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686: No bootstrap required, opened a new log
I20260620 14:24:47.022681 8410 ts_tablet_manager.cc:1397] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:47.022827 8410 raft_consensus.cc:348] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.022907 8410 raft_consensus.cc:374] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.022936 8410 raft_consensus.cc:729] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9be9ff83c14404aa332f9362c43686, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.022687 8407 tablet_bootstrap.cc:654] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.022979 8410 consensus_queue.cc:260] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.023066 8410 ts_tablet_manager.cc:1428] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:47.023157 8409 tablet_bootstrap.cc:492] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b: No bootstrap required, opened a new log
I20260620 14:24:47.023213 8409 ts_tablet_manager.cc:1397] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:47.022547 8430 tablet_replica.cc:330] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467: stopping tablet replica
I20260620 14:24:47.023339 8430 raft_consensus.cc:2229] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:47.023358 8409 raft_consensus.cc:348] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.023417 8409 raft_consensus.cc:374] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.023437 8409 raft_consensus.cc:729] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20c036d76804f5785c139c24717477b, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.023448 8430 raft_consensus.cc:2258] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:47.023487 8409 consensus_queue.cc:260] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.023571 8409 ts_tablet_manager.cc:1428] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:47.023922 8430 ts_tablet_manager.cc:1905] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:47.025185 8430 ts_tablet_manager.cc:1918] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:47.025280 8430 log.cc:1196] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/ba07b19fa38f40f1b5a92fddcb77ab6c
I20260620 14:24:47.025549 8430 ts_tablet_manager.cc:1939] T ba07b19fa38f40f1b5a92fddcb77ab6c P 310d2dad493340c8a942642c21ed2467: Deleting consensus metadata
I20260620 14:24:47.025960 7880 catalog_manager.cc:4841] TS 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429): tablet ba07b19fa38f40f1b5a92fddcb77ab6c (table test-table [id=663ffc94a56f4f9cbbb5b6ac022dc31a]) successfully deleted
I20260620 14:24:47.026073 8407 tablet_bootstrap.cc:492] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467: No bootstrap required, opened a new log
I20260620 14:24:47.026113 8407 ts_tablet_manager.cc:1397] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 14:24:47.026275 8407 raft_consensus.cc:348] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.026326 8407 raft_consensus.cc:374] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.026414 8407 raft_consensus.cc:729] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310d2dad493340c8a942642c21ed2467, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.026460 8407 consensus_queue.cc:260] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.026562 8407 ts_tablet_manager.cc:1428] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260620 14:24:47.067567 8380 tablet.cc:2369] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:24:47.131089 8415 raft_consensus.cc:482] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:47.131240 8415 raft_consensus.cc:504] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.131645 8415 leader_election.cc:288] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a20c036d76804f5785c139c24717477b (127.0.98.1:39123), 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429)
I20260620 14:24:47.135170 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.132025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54451 (local address 127.0.98.1:39123)
0620 14:24:47.132082 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.132084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.132092 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:47.132122 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.132123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.132132 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.132185 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:47.132355 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.133020 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.134425 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.134764 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.134822 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.134832 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.135039 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.135097 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:24:47.135166 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.131908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46661 (local address 127.0.98.3:44429)
0620 14:24:47.131963 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.131965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.132038 (+ 73us) server_negotiation.cc:407] Connection header received
0620 14:24:47.132156 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.132157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.132170 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.132221 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:24:47.132397 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.132867 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.134405 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.134779 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.134879 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.134890 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.135068 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.135106 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:24:47.135166 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.131834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39123 (local address 127.0.98.2:54451)
0620 14:24:47.131933 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:47.131946 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:47.131992 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:47.132198 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:47.132199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:47.132201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:47.132274 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.132276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.133113 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.133113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.134252 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.134256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.134861 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.134862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.134953 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.134959 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:24:47.135012 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:24:47.135077 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":3}
I20260620 14:24:47.135306 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.131848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44429 (local address 127.0.98.2:46661)
0620 14:24:47.131973 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:47.131983 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:47.132066 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:47.132243 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:47.132244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:47.132246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:24:47.132334 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.132336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.132996 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.132997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.134252 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.134256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.134917 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.134918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.134972 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.134975 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:47.135052 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:24:47.135094 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":11,"thread_start_us":62,"threads_started":1}
I20260620 14:24:47.135501 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad6424e365b4263821c7af56edb3523" candidate_uuid: "1e9be9ff83c14404aa332f9362c43686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20c036d76804f5785c139c24717477b" is_pre_election: true
I20260620 14:24:47.135643 8053 raft_consensus.cc:2454] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e9be9ff83c14404aa332f9362c43686 in term 0.
I20260620 14:24:47.135795 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad6424e365b4263821c7af56edb3523" candidate_uuid: "1e9be9ff83c14404aa332f9362c43686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310d2dad493340c8a942642c21ed2467" is_pre_election: true
I20260620 14:24:47.135841 8126 leader_election.cc:302] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9be9ff83c14404aa332f9362c43686, a20c036d76804f5785c139c24717477b; no voters:
I20260620 14:24:47.135900 8333 raft_consensus.cc:2454] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e9be9ff83c14404aa332f9362c43686 in term 0.
I20260620 14:24:47.135986 8415 raft_consensus.cc:2781] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:47.136034 8415 raft_consensus.cc:482] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:47.136059 8415 raft_consensus.cc:3037] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.136797 8415 raft_consensus.cc:504] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.136957 8415 leader_election.cc:288] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [CANDIDATE]: Term 1 election: Requested vote from peers a20c036d76804f5785c139c24717477b (127.0.98.1:39123), 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429)
I20260620 14:24:47.137089 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad6424e365b4263821c7af56edb3523" candidate_uuid: "1e9be9ff83c14404aa332f9362c43686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20c036d76804f5785c139c24717477b"
I20260620 14:24:47.137166 8053 raft_consensus.cc:3037] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.137109 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad6424e365b4263821c7af56edb3523" candidate_uuid: "1e9be9ff83c14404aa332f9362c43686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310d2dad493340c8a942642c21ed2467"
I20260620 14:24:47.137234 8333 raft_consensus.cc:3037] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.137881 8333 raft_consensus.cc:2454] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e9be9ff83c14404aa332f9362c43686 in term 1.
I20260620 14:24:47.137915 8053 raft_consensus.cc:2454] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e9be9ff83c14404aa332f9362c43686 in term 1.
I20260620 14:24:47.138134 8126 leader_election.cc:302] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9be9ff83c14404aa332f9362c43686, a20c036d76804f5785c139c24717477b; no voters:
I20260620 14:24:47.138303 8415 raft_consensus.cc:2781] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:47.138489 8415 raft_consensus.cc:686] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 1 LEADER]: Becoming Leader. State: Replica: 1e9be9ff83c14404aa332f9362c43686, State: Running, Role: LEADER
I20260620 14:24:47.138581 8415 consensus_queue.cc:237] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.139433 7895 catalog_manager.cc:5495] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1e9be9ff83c14404aa332f9362c43686" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:24:47 dist-test-slave-hfc3 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965486, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:47.190112 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.183079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34115 (local address 127.0.0.1:51466)
0620 14:24:47.183244 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:47.183257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:47.183351 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:47.183516 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:47.183517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:47.183946 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:47.184006 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.184009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.184692 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.184694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.185223 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.185224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.185602 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.185603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.185655 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.186182 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:47.186186 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:47.187606 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:47.188976 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:47.188984 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:47.188986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:47.189174 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:47.189386 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:47.189387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:47.189387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:47.189445 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:47.189682 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:47.189689 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:47.189735 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:24:47.189820 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:24:47.189970 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260620 14:24:47.190114 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.183122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51466 (local address 127.0.98.62:34115)
0620 14:24:47.183187 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.183189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.183351 (+ 162us) server_negotiation.cc:407] Connection header received
0620 14:24:47.183431 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.183431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.183448 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.183512 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:47.184113 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.184605 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.185318 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.185501 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.185588 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.187696 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:47.187697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:47.187698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:47.187705 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:47.188881 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:47.189258 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:47.189259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:47.189259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:47.189313 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:47.189506 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:47.189507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:47.189508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:47.189617 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:47.189691 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.189842 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.190024 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":36}
I20260620 14:24:47.192310 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51466:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:24:47.192636 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:47.194135 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 5bdb89913dc143ecbb8a012ad3eb4b0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d066363cb1f4dab94194014b14d56d3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:24:47.194135 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 5bdb89913dc143ecbb8a012ad3eb4b0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d066363cb1f4dab94194014b14d56d3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:24:47.194160 8313 tablet_service.cc:1467] Processing CreateTablet for tablet 5bdb89913dc143ecbb8a012ad3eb4b0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d066363cb1f4dab94194014b14d56d3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:24:47.194296 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bdb89913dc143ecbb8a012ad3eb4b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.194296 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bdb89913dc143ecbb8a012ad3eb4b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.194315 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bdb89913dc143ecbb8a012ad3eb4b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.195779 8410 tablet_bootstrap.cc:492] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686: Bootstrap starting.
I20260620 14:24:47.195779 8409 tablet_bootstrap.cc:492] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: Bootstrap starting.
I20260620 14:24:47.195807 8407 tablet_bootstrap.cc:492] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467: Bootstrap starting.
I20260620 14:24:47.196142 8407 tablet_bootstrap.cc:654] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.196143 8409 tablet_bootstrap.cc:654] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.196151 8410 tablet_bootstrap.cc:654] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.196761 8407 tablet_bootstrap.cc:492] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467: No bootstrap required, opened a new log
I20260620 14:24:47.196818 8407 ts_tablet_manager.cc:1397] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:47.196858 8410 tablet_bootstrap.cc:492] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686: No bootstrap required, opened a new log
I20260620 14:24:47.196902 8410 ts_tablet_manager.cc:1397] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:47.196946 8409 tablet_bootstrap.cc:492] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: No bootstrap required, opened a new log
I20260620 14:24:47.196990 8409 ts_tablet_manager.cc:1397] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:47.196971 8407 raft_consensus.cc:348] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.197022 8407 raft_consensus.cc:374] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.197036 8407 raft_consensus.cc:729] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310d2dad493340c8a942642c21ed2467, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.197054 8410 raft_consensus.cc:348] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.197119 8410 raft_consensus.cc:374] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.197084 8409 raft_consensus.cc:348] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.197105 8407 consensus_queue.cc:260] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.197142 8410 raft_consensus.cc:729] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9be9ff83c14404aa332f9362c43686, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.197144 8409 raft_consensus.cc:374] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.197165 8409 raft_consensus.cc:729] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20c036d76804f5785c139c24717477b, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.197203 8407 ts_tablet_manager.cc:1428] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:47.197188 8410 consensus_queue.cc:260] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.197206 8409 consensus_queue.cc:260] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.197284 8410 ts_tablet_manager.cc:1428] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:47.197284 8409 ts_tablet_manager.cc:1428] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:47.197269 8413 tablet_replica.cc:439] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } }
I20260620 14:24:47.197361 8413 tablet_replica.cc:442] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:47.197381 8417 tablet_replica.cc:439] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } }
I20260620 14:24:47.197422 8415 tablet_replica.cc:439] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } }
I20260620 14:24:47.197454 8417 tablet_replica.cc:442] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:47.197468 8415 tablet_replica.cc:442] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:47.277837 8417 raft_consensus.cc:482] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:47.277963 8417 raft_consensus.cc:504] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.278095 8417 leader_election.cc:288] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2:34095), 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429)
I20260620 14:24:47.278535 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bdb89913dc143ecbb8a012ad3eb4b0f" candidate_uuid: "a20c036d76804f5785c139c24717477b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9be9ff83c14404aa332f9362c43686" is_pre_election: true
I20260620 14:24:47.278535 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bdb89913dc143ecbb8a012ad3eb4b0f" candidate_uuid: "a20c036d76804f5785c139c24717477b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310d2dad493340c8a942642c21ed2467" is_pre_election: true
I20260620 14:24:47.278650 8333 raft_consensus.cc:2454] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20c036d76804f5785c139c24717477b in term 0.
I20260620 14:24:47.278650 8193 raft_consensus.cc:2454] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20c036d76804f5785c139c24717477b in term 0.
I20260620 14:24:47.278815 7985 leader_election.cc:302] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 310d2dad493340c8a942642c21ed2467, a20c036d76804f5785c139c24717477b; no voters:
I20260620 14:24:47.278921 8417 raft_consensus.cc:2781] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:47.278961 8417 raft_consensus.cc:482] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:47.278981 8417 raft_consensus.cc:3037] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.279668 8417 raft_consensus.cc:504] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.279796 8417 leader_election.cc:288] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [CANDIDATE]: Term 1 election: Requested vote from peers 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2:34095), 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429)
I20260620 14:24:47.279987 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bdb89913dc143ecbb8a012ad3eb4b0f" candidate_uuid: "a20c036d76804f5785c139c24717477b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310d2dad493340c8a942642c21ed2467"
I20260620 14:24:47.280002 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bdb89913dc143ecbb8a012ad3eb4b0f" candidate_uuid: "a20c036d76804f5785c139c24717477b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9be9ff83c14404aa332f9362c43686"
I20260620 14:24:47.280067 8333 raft_consensus.cc:3037] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.280088 8193 raft_consensus.cc:3037] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.280711 8193 raft_consensus.cc:2454] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20c036d76804f5785c139c24717477b in term 1.
I20260620 14:24:47.280709 8333 raft_consensus.cc:2454] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20c036d76804f5785c139c24717477b in term 1.
I20260620 14:24:47.280889 7985 leader_election.cc:302] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9be9ff83c14404aa332f9362c43686, a20c036d76804f5785c139c24717477b; no voters:
I20260620 14:24:47.281005 8417 raft_consensus.cc:2781] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:47.281072 8417 raft_consensus.cc:686] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [term 1 LEADER]: Becoming Leader. State: Replica: a20c036d76804f5785c139c24717477b, State: Running, Role: LEADER
I20260620 14:24:47.281124 8417 consensus_queue.cc:237] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.281436 8426 tablet_replica.cc:439] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: TxnStatusTablet state changed. Reason: New leader a20c036d76804f5785c139c24717477b. Latest consensus state: current_term: 1 leader_uuid: "a20c036d76804f5785c139c24717477b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } }
I20260620 14:24:47.281529 8426 tablet_replica.cc:442] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:24:47.281746 8449 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:24:47.281771 7895 catalog_manager.cc:5495] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b reported cstate change: term changed from 0 to 1, leader changed from <none> to a20c036d76804f5785c139c24717477b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a20c036d76804f5785c139c24717477b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } health_report { overall_health: UNKNOWN } } }
I20260620 14:24:47.281901 8449 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:24:47.294492 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.291835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.0.98.1:39123)
0620 14:24:47.291877 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.291879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.291917 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:24:47.292008 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.292024 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.292038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.292101 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:47.292213 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.292670 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.293764 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.293958 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.294015 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.294215 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:47.294243 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:47.294282 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.294395 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.294418 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:24:47.294526 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.291794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39123 (local address 127.0.0.1:41656)
0620 14:24:47.291873 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:47.291884 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:47.291936 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:47.292104 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:47.292105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:47.292106 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:24:47.292161 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.292164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.292746 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.292747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.293670 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.293674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.294038 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.294039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.294107 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.294128 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:24:47.294302 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:24:47.294310 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:24:47.294382 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:24:47.294441 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:24:47.295547 8193 raft_consensus.cc:1261] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Refusing update from remote peer a20c036d76804f5785c139c24717477b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:47.295562 8333 raft_consensus.cc:1261] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Refusing update from remote peer a20c036d76804f5785c139c24717477b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:47.295781 8417 consensus_queue.cc:1035] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:47.295917 8426 consensus_queue.cc:1035] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b [LEADER]: Connected to new peer: Peer: permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:47.296430 8413 tablet_replica.cc:439] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467: TxnStatusTablet state changed. Reason: New leader a20c036d76804f5785c139c24717477b. Latest consensus state: current_term: 1 leader_uuid: "a20c036d76804f5785c139c24717477b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } }
I20260620 14:24:47.296520 8413 tablet_replica.cc:442] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:47.296614 8415 tablet_replica.cc:439] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686: TxnStatusTablet state changed. Reason: New leader a20c036d76804f5785c139c24717477b. Latest consensus state: current_term: 1 leader_uuid: "a20c036d76804f5785c139c24717477b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } }
I20260620 14:24:47.296697 8415 tablet_replica.cc:442] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:47.297317 8426 tablet_replica.cc:439] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a20c036d76804f5785c139c24717477b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } }
I20260620 14:24:47.297329 8414 tablet_replica.cc:439] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a20c036d76804f5785c139c24717477b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } }
I20260620 14:24:47.297497 8414 tablet_replica.cc:442] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:24:47.297406 8426 tablet_replica.cc:442] T 5bdb89913dc143ecbb8a012ad3eb4b0f P a20c036d76804f5785c139c24717477b: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:24:47.297741 8413 tablet_replica.cc:439] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a20c036d76804f5785c139c24717477b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } }
I20260620 14:24:47.297832 8413 tablet_replica.cc:442] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 310d2dad493340c8a942642c21ed2467: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:47.298445 8415 tablet_replica.cc:439] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a20c036d76804f5785c139c24717477b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } } }
I20260620 14:24:47.298534 8415 tablet_replica.cc:442] T 5bdb89913dc143ecbb8a012ad3eb4b0f P 1e9be9ff83c14404aa332f9362c43686: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:24:47.302121 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.299547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.0.98.2:34095)
0620 14:24:47.299584 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.299586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.299638 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:24:47.299719 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.299719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.299734 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.299793 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:47.299914 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.300370 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.301414 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.301588 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.301639 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.301798 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:47.301898 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:47.301952 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.302017 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.302033 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:24:47.307574 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.304728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47699 (local address 127.0.98.1:39123)
0620 14:24:47.304811 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.304813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.304875 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:24:47.304961 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.304962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.304974 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.305030 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:47.305184 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.305650 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.306513 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.306717 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.306863 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.307221 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:47.307264 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:47.307329 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.307475 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.307508 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:24:47.307577 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.304661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39123 (local address 127.0.98.2:47699)
0620 14:24:47.304815 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:47.304827 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:47.304893 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:47.305033 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:47.305034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:47.305035 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:24:47.305096 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.305098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.305738 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.305739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.306422 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.306425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.306863 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.306864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.307066 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.307082 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:24:47.307366 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:24:47.307369 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:24:47.307434 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:24:47.307483 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260620 14:24:47.310024 8333 raft_consensus.cc:1261] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Refusing update from remote peer 1e9be9ff83c14404aa332f9362c43686: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:47.310087 8053 raft_consensus.cc:1261] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Refusing update from remote peer 1e9be9ff83c14404aa332f9362c43686: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:47.310305 8415 consensus_queue.cc:1035] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [LEADER]: Connected to new peer: Peer: permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:47.310487 8437 consensus_queue.cc:1035] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:47.321866 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.318974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34095 (local address 127.0.98.1:42627)
0620 14:24:47.319121 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:47.319138 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:47.319214 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:47.319370 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:47.319371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:47.319373 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:24:47.319426 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.319427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.320086 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.320087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.321056 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.321058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.321431 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.321432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.321489 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.321504 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:24:47.321727 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:24:47.321729 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:24:47.321762 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 14:24:47.321799 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260620 14:24:47.321870 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.319008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42627 (local address 127.0.98.2:34095)
0620 14:24:47.319046 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.319048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.319196 (+ 148us) server_negotiation.cc:407] Connection header received
0620 14:24:47.319265 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.319266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.319277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.319356 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:47.319514 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.319997 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.321135 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.321346 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.321433 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.321615 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:47.321656 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:47.321721 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.321774 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.321793 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:24:47.331715 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:47.331840 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:47.332448 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c: Sending DeleteTablet for 3 replicas of tablet 3ad6424e365b4263821c7af56edb3523
I20260620 14:24:47.332708 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad6424e365b4263821c7af56edb3523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41636
I20260620 14:24:47.332784 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad6424e365b4263821c7af56edb3523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52884
I20260620 14:24:47.332819 8432 tablet_replica.cc:330] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b: stopping tablet replica
I20260620 14:24:47.332894 8432 raft_consensus.cc:2229] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:47.332899 8430 tablet_replica.cc:330] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467: stopping tablet replica
I20260620 14:24:47.332939 8432 raft_consensus.cc:2258] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:47.332959 8430 raft_consensus.cc:2229] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:47.332996 8430 raft_consensus.cc:2258] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:47.333259 8432 ts_tablet_manager.cc:1905] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:47.333340 8430 ts_tablet_manager.cc:1905] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:47.334880 8432 ts_tablet_manager.cc:1918] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:47.334941 8432 log.cc:1196] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/wal/wals/3ad6424e365b4263821c7af56edb3523
I20260620 14:24:47.335160 8432 ts_tablet_manager.cc:1939] T 3ad6424e365b4263821c7af56edb3523 P a20c036d76804f5785c139c24717477b: Deleting consensus metadata
I20260620 14:24:47.335246 8430 ts_tablet_manager.cc:1918] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:47.335306 8430 log.cc:1196] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/3ad6424e365b4263821c7af56edb3523
I20260620 14:24:47.335532 8430 ts_tablet_manager.cc:1939] T 3ad6424e365b4263821c7af56edb3523 P 310d2dad493340c8a942642c21ed2467: Deleting consensus metadata
I20260620 14:24:47.335927 7881 catalog_manager.cc:4841] TS a20c036d76804f5785c139c24717477b (127.0.98.1:39123): tablet 3ad6424e365b4263821c7af56edb3523 (table test-table [id=d3e2c66b56194b26abd2a25adee68b87]) successfully deleted
I20260620 14:24:47.336021 7880 catalog_manager.cc:4841] TS 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429): tablet 3ad6424e365b4263821c7af56edb3523 (table test-table [id=d3e2c66b56194b26abd2a25adee68b87]) successfully deleted
I20260620 14:24:47.336443 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad6424e365b4263821c7af56edb3523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51320
I20260620 14:24:47.336562 8431 tablet_replica.cc:330] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686: stopping tablet replica
I20260620 14:24:47.336615 8431 raft_consensus.cc:2229] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:47.336709 8431 raft_consensus.cc:2258] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:47.336957 8431 ts_tablet_manager.cc:1905] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:47.338203 8431 ts_tablet_manager.cc:1918] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:47.338308 8431 log.cc:1196] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/3ad6424e365b4263821c7af56edb3523
I20260620 14:24:47.338544 8431 ts_tablet_manager.cc:1939] T 3ad6424e365b4263821c7af56edb3523 P 1e9be9ff83c14404aa332f9362c43686: Deleting consensus metadata
I20260620 14:24:47.338872 7880 catalog_manager.cc:4841] TS 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2:34095): tablet 3ad6424e365b4263821c7af56edb3523 (table test-table [id=d3e2c66b56194b26abd2a25adee68b87]) successfully deleted
Jun 20 14:24:47 dist-test-slave-hfc3 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:24:47 dist-test-slave-hfc3 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:47 dist-test-slave-hfc3 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:24:47.367564 7944 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:47.361471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.0.98.62:34115)
0620 14:24:47.361527 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.361530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.361679 (+ 149us) server_negotiation.cc:407] Connection header received
0620 14:24:47.361770 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.361771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.361782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.361858 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:47.362511 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.363098 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.364138 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.364364 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.364450 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.367512 (+ 3062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51480: BlockingRecv error: recv got EOF from 127.0.0.1:51480 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 14:24:47 dist-test-slave-hfc3 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:24:47.376067 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.368682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.0.98.62:34115)
0620 14:24:47.368719 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.368721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.368963 (+ 242us) server_negotiation.cc:407] Connection header received
0620 14:24:47.369038 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.369039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.369049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.369106 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:47.369672 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.370262 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.371023 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.371226 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.371311 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.373457 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:47.373459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:47.373459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:47.373468 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:47.374908 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:47.375278 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:47.375280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:47.375280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:47.375348 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:47.375553 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:47.375554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:47.375554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:47.375667 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:47.375771 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.375877 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.375953 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:24:47.377689 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:24:47.377882 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:47.379190 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 36ca60d5006248cda34855b2e45a5537 (DEFAULT_TABLE table=test-table [id=a7eb97adaa544c8d8ee339fe9724366f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:47.379262 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 36ca60d5006248cda34855b2e45a5537 (DEFAULT_TABLE table=test-table [id=a7eb97adaa544c8d8ee339fe9724366f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:47.379326 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ca60d5006248cda34855b2e45a5537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.379366 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ca60d5006248cda34855b2e45a5537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.380676 8313 tablet_service.cc:1467] Processing CreateTablet for tablet 36ca60d5006248cda34855b2e45a5537 (DEFAULT_TABLE table=test-table [id=a7eb97adaa544c8d8ee339fe9724366f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:47.380704 8410 tablet_bootstrap.cc:492] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686: Bootstrap starting.
I20260620 14:24:47.380795 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ca60d5006248cda34855b2e45a5537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.381167 8410 tablet_bootstrap.cc:654] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.381839 8410 tablet_bootstrap.cc:492] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686: No bootstrap required, opened a new log
I20260620 14:24:47.381881 8410 ts_tablet_manager.cc:1397] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:47.382020 8407 tablet_bootstrap.cc:492] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467: Bootstrap starting.
I20260620 14:24:47.382055 8409 tablet_bootstrap.cc:492] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b: Bootstrap starting.
I20260620 14:24:47.382037 8410 raft_consensus.cc:348] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.382105 8410 raft_consensus.cc:374] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.382125 8410 raft_consensus.cc:729] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9be9ff83c14404aa332f9362c43686, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.382171 8410 consensus_queue.cc:260] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.382251 8410 ts_tablet_manager.cc:1428] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:47.382488 8407 tablet_bootstrap.cc:654] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.382515 8409 tablet_bootstrap.cc:654] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.383468 8409 tablet_bootstrap.cc:492] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b: No bootstrap required, opened a new log
I20260620 14:24:47.383514 8409 ts_tablet_manager.cc:1397] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:47.383654 8409 raft_consensus.cc:348] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.383718 8409 raft_consensus.cc:374] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.383739 8409 raft_consensus.cc:729] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20c036d76804f5785c139c24717477b, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.383780 8409 consensus_queue.cc:260] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.383908 8409 ts_tablet_manager.cc:1428] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:47.384081 8407 tablet_bootstrap.cc:492] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467: No bootstrap required, opened a new log
I20260620 14:24:47.384124 8407 ts_tablet_manager.cc:1397] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:47.384279 8407 raft_consensus.cc:348] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.384372 8407 raft_consensus.cc:374] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.384402 8407 raft_consensus.cc:729] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310d2dad493340c8a942642c21ed2467, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.384444 8407 consensus_queue.cc:260] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.384536 8407 ts_tablet_manager.cc:1428] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:47.586210 8459 raft_consensus.cc:482] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:47.586315 8459 raft_consensus.cc:504] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.586568 8459 leader_election.cc:288] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a20c036d76804f5785c139c24717477b (127.0.98.1:39123), 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429)
I20260620 14:24:47.586808 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ca60d5006248cda34855b2e45a5537" candidate_uuid: "1e9be9ff83c14404aa332f9362c43686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310d2dad493340c8a942642c21ed2467" is_pre_election: true
I20260620 14:24:47.586799 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ca60d5006248cda34855b2e45a5537" candidate_uuid: "1e9be9ff83c14404aa332f9362c43686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20c036d76804f5785c139c24717477b" is_pre_election: true
I20260620 14:24:47.586906 8333 raft_consensus.cc:2454] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e9be9ff83c14404aa332f9362c43686 in term 0.
I20260620 14:24:47.586933 8053 raft_consensus.cc:2454] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e9be9ff83c14404aa332f9362c43686 in term 0.
I20260620 14:24:47.587059 8125 leader_election.cc:302] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9be9ff83c14404aa332f9362c43686, 310d2dad493340c8a942642c21ed2467; no voters:
I20260620 14:24:47.587190 8459 raft_consensus.cc:2781] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:47.587241 8459 raft_consensus.cc:482] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:47.587256 8459 raft_consensus.cc:3037] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.588016 8459 raft_consensus.cc:504] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.588156 8459 leader_election.cc:288] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [CANDIDATE]: Term 1 election: Requested vote from peers a20c036d76804f5785c139c24717477b (127.0.98.1:39123), 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429)
I20260620 14:24:47.588344 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ca60d5006248cda34855b2e45a5537" candidate_uuid: "1e9be9ff83c14404aa332f9362c43686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310d2dad493340c8a942642c21ed2467"
I20260620 14:24:47.588343 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ca60d5006248cda34855b2e45a5537" candidate_uuid: "1e9be9ff83c14404aa332f9362c43686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20c036d76804f5785c139c24717477b"
I20260620 14:24:47.588431 8333 raft_consensus.cc:3037] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.588433 8053 raft_consensus.cc:3037] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.589244 8053 raft_consensus.cc:2454] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e9be9ff83c14404aa332f9362c43686 in term 1.
I20260620 14:24:47.589244 8333 raft_consensus.cc:2454] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e9be9ff83c14404aa332f9362c43686 in term 1.
I20260620 14:24:47.589429 8125 leader_election.cc:302] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9be9ff83c14404aa332f9362c43686, 310d2dad493340c8a942642c21ed2467; no voters:
I20260620 14:24:47.589531 8459 raft_consensus.cc:2781] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:47.589584 8459 raft_consensus.cc:686] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 1 LEADER]: Becoming Leader. State: Replica: 1e9be9ff83c14404aa332f9362c43686, State: Running, Role: LEADER
I20260620 14:24:47.589658 8459 consensus_queue.cc:237] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.590271 7894 catalog_manager.cc:5495] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1e9be9ff83c14404aa332f9362c43686" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } health_report { overall_health: UNKNOWN } } }
I20260620 14:24:47.610612 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.608284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51342 (local address 127.0.98.2:34095)
0620 14:24:47.608356 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.608358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.608368 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:47.608395 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.608396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.608407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.608485 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:47.608610 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.609091 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.609888 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.610079 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.610157 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.610313 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:47.610369 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:47.610436 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.610491 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.610515 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:24:47.611495 8333 raft_consensus.cc:1261] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Refusing update from remote peer 1e9be9ff83c14404aa332f9362c43686: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:47.611747 8437 consensus_queue.cc:1035] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [LEADER]: Connected to new peer: Peer: permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:47.611932 8053 raft_consensus.cc:1261] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Refusing update from remote peer 1e9be9ff83c14404aa332f9362c43686: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:47.612201 8437 consensus_queue.cc:1035] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:47.614974 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:47.615064 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:47.615736 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c: Sending DeleteTablet for 3 replicas of tablet 36ca60d5006248cda34855b2e45a5537
I20260620 14:24:47.615993 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 36ca60d5006248cda34855b2e45a5537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51320
I20260620 14:24:47.616046 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet 36ca60d5006248cda34855b2e45a5537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52884
I20260620 14:24:47.616077 8431 tablet_replica.cc:330] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686: stopping tablet replica
I20260620 14:24:47.616127 8431 raft_consensus.cc:2229] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:47.616190 8431 raft_consensus.cc:2258] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:47.616258 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet 36ca60d5006248cda34855b2e45a5537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41636
I20260620 14:24:47.616433 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:24:47.616385 8430 tablet_replica.cc:330] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467: stopping tablet replica
I20260620 14:24:47.616508 8431 ts_tablet_manager.cc:1905] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:24:47.616567 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:24:47.616556 8430 raft_consensus.cc:2229] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:47.616659 8430 raft_consensus.cc:2258] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:47.616801 8432 tablet_replica.cc:330] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b: stopping tablet replica
I20260620 14:24:47.616858 8432 raft_consensus.cc:2229] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:47.616890 8432 raft_consensus.cc:2258] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:47.617231 8432 ts_tablet_manager.cc:1905] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:47.617435 8430 ts_tablet_manager.cc:1905] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:47.618057 8431 ts_tablet_manager.cc:1918] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:47.618108 8431 log.cc:1196] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/36ca60d5006248cda34855b2e45a5537
I20260620 14:24:47.618366 8431 ts_tablet_manager.cc:1939] T 36ca60d5006248cda34855b2e45a5537 P 1e9be9ff83c14404aa332f9362c43686: Deleting consensus metadata
I20260620 14:24:47.618851 8432 ts_tablet_manager.cc:1918] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:47.618892 8432 log.cc:1196] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-0/wal/wals/36ca60d5006248cda34855b2e45a5537
I20260620 14:24:47.618961 8430 ts_tablet_manager.cc:1918] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:24:47.619015 8430 log.cc:1196] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/36ca60d5006248cda34855b2e45a5537
I20260620 14:24:47.619042 8432 ts_tablet_manager.cc:1939] T 36ca60d5006248cda34855b2e45a5537 P a20c036d76804f5785c139c24717477b: Deleting consensus metadata
I20260620 14:24:47.619267 8430 ts_tablet_manager.cc:1939] T 36ca60d5006248cda34855b2e45a5537 P 310d2dad493340c8a942642c21ed2467: Deleting consensus metadata
I20260620 14:24:47.619478 7881 catalog_manager.cc:4841] TS a20c036d76804f5785c139c24717477b (127.0.98.1:39123): tablet 36ca60d5006248cda34855b2e45a5537 (table test-table [id=a7eb97adaa544c8d8ee339fe9724366f]) successfully deleted
I20260620 14:24:47.619612 7880 catalog_manager.cc:4841] TS 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429): tablet 36ca60d5006248cda34855b2e45a5537 (table test-table [id=a7eb97adaa544c8d8ee339fe9724366f]) successfully deleted
I20260620 14:24:47.619889 7880 catalog_manager.cc:4841] TS 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2:34095): tablet 36ca60d5006248cda34855b2e45a5537 (table test-table [id=a7eb97adaa544c8d8ee339fe9724366f]) successfully deleted
I20260620 14:24:47.620013 8173 tablet_service.cc:1467] Processing CreateTablet for tablet a0365de20e4642c398dc9a9c1300cff0 (DEFAULT_TABLE table=test-table [id=41f90a956d2b47e0bae59c5f2c045884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:47.620074 8033 tablet_service.cc:1467] Processing CreateTablet for tablet a0365de20e4642c398dc9a9c1300cff0 (DEFAULT_TABLE table=test-table [id=41f90a956d2b47e0bae59c5f2c045884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:47.620124 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0365de20e4642c398dc9a9c1300cff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.620106 8313 tablet_service.cc:1467] Processing CreateTablet for tablet a0365de20e4642c398dc9a9c1300cff0 (DEFAULT_TABLE table=test-table [id=41f90a956d2b47e0bae59c5f2c045884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:24:47.620179 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0365de20e4642c398dc9a9c1300cff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.620194 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0365de20e4642c398dc9a9c1300cff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:47.621127 8409 tablet_bootstrap.cc:492] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b: Bootstrap starting.
I20260620 14:24:47.621284 8407 tablet_bootstrap.cc:492] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467: Bootstrap starting.
I20260620 14:24:47.621299 8410 tablet_bootstrap.cc:492] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686: Bootstrap starting.
I20260620 14:24:47.621475 8409 tablet_bootstrap.cc:654] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.621613 8410 tablet_bootstrap.cc:654] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.621991 8407 tablet_bootstrap.cc:654] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:47.622184 8410 tablet_bootstrap.cc:492] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686: No bootstrap required, opened a new log
I20260620 14:24:47.622238 8410 ts_tablet_manager.cc:1397] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:47.622382 8410 raft_consensus.cc:348] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.622473 8410 raft_consensus.cc:374] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.622498 8410 raft_consensus.cc:729] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9be9ff83c14404aa332f9362c43686, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.622553 8410 consensus_queue.cc:260] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.622654 8407 tablet_bootstrap.cc:492] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467: No bootstrap required, opened a new log
I20260620 14:24:47.622663 8410 ts_tablet_manager.cc:1428] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:47.622704 8407 ts_tablet_manager.cc:1397] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:47.622862 8407 raft_consensus.cc:348] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.622932 8407 raft_consensus.cc:374] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.622952 8407 raft_consensus.cc:729] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310d2dad493340c8a942642c21ed2467, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.622990 8407 consensus_queue.cc:260] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.623112 8407 ts_tablet_manager.cc:1428] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:47.623466 8409 tablet_bootstrap.cc:492] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b: No bootstrap required, opened a new log
I20260620 14:24:47.623520 8409 ts_tablet_manager.cc:1397] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:24:47.623595 8409 raft_consensus.cc:348] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.623643 8409 raft_consensus.cc:374] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:47.623656 8409 raft_consensus.cc:729] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20c036d76804f5785c139c24717477b, State: Initialized, Role: FOLLOWER
I20260620 14:24:47.623711 8409 consensus_queue.cc:260] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.623800 8409 ts_tablet_manager.cc:1428] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:47.655966 8414 raft_consensus.cc:482] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:24:47.656076 8414 raft_consensus.cc:504] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.656268 8414 leader_election.cc:288] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2:34095), 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429)
I20260620 14:24:47.656478 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0365de20e4642c398dc9a9c1300cff0" candidate_uuid: "a20c036d76804f5785c139c24717477b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9be9ff83c14404aa332f9362c43686" is_pre_election: true
I20260620 14:24:47.656512 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0365de20e4642c398dc9a9c1300cff0" candidate_uuid: "a20c036d76804f5785c139c24717477b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310d2dad493340c8a942642c21ed2467" is_pre_election: true
I20260620 14:24:47.656603 8333 raft_consensus.cc:2454] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20c036d76804f5785c139c24717477b in term 0.
I20260620 14:24:47.656600 8193 raft_consensus.cc:2454] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20c036d76804f5785c139c24717477b in term 0.
I20260620 14:24:47.656773 7985 leader_election.cc:302] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9be9ff83c14404aa332f9362c43686, a20c036d76804f5785c139c24717477b; no voters:
I20260620 14:24:47.656867 8414 raft_consensus.cc:2781] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:24:47.656914 8414 raft_consensus.cc:482] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:24:47.656936 8414 raft_consensus.cc:3037] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.657711 8414 raft_consensus.cc:504] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.657842 8414 leader_election.cc:288] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [CANDIDATE]: Term 1 election: Requested vote from peers 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2:34095), 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429)
I20260620 14:24:47.657980 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0365de20e4642c398dc9a9c1300cff0" candidate_uuid: "a20c036d76804f5785c139c24717477b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9be9ff83c14404aa332f9362c43686"
I20260620 14:24:47.658074 8193 raft_consensus.cc:3037] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.658041 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0365de20e4642c398dc9a9c1300cff0" candidate_uuid: "a20c036d76804f5785c139c24717477b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310d2dad493340c8a942642c21ed2467"
I20260620 14:24:47.658138 8333 raft_consensus.cc:3037] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:47.658761 8193 raft_consensus.cc:2454] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20c036d76804f5785c139c24717477b in term 1.
I20260620 14:24:47.658828 8333 raft_consensus.cc:2454] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20c036d76804f5785c139c24717477b in term 1.
I20260620 14:24:47.658934 7985 leader_election.cc:302] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9be9ff83c14404aa332f9362c43686, a20c036d76804f5785c139c24717477b; no voters:
I20260620 14:24:47.659035 8414 raft_consensus.cc:2781] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:47.659086 8414 raft_consensus.cc:686] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 1 LEADER]: Becoming Leader. State: Replica: a20c036d76804f5785c139c24717477b, State: Running, Role: LEADER
I20260620 14:24:47.659142 8414 consensus_queue.cc:237] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } }
I20260620 14:24:47.659672 7894 catalog_manager.cc:5495] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b reported cstate change: term changed from 0 to 1, leader changed from <none> to a20c036d76804f5785c139c24717477b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a20c036d76804f5785c139c24717477b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20c036d76804f5785c139c24717477b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 } health_report { overall_health: UNKNOWN } } }
I20260620 14:24:47.672492 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.670252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.0.98.1:39123)
0620 14:24:47.670296 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.670298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.670307 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:24:47.670411 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.670412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.670423 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.670479 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:47.670582 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.671077 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.671794 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.672004 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.672088 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.672236 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:47.672270 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:47.672321 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.672392 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.672411 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:24:47.676223 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.674327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33311 (local address 127.0.98.1:39123)
0620 14:24:47.674374 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:47.674376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:47.674424 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:24:47.674476 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:47.674477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:47.674484 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:47.674526 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:24:47.674639 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.675057 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.675697 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.675865 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.675928 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.676039 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:24:47.676061 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:24:47.676097 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:47.676144 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 14:24:47.676157 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:24:47.676285 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.674276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39123 (local address 127.0.98.1:33311)
0620 14:24:47.674369 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:47.674380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:47.674430 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:47.674549 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:47.674550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:47.674551 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:24:47.674597 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.674598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.675110 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.675111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.675628 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:47.675629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:47.675936 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:47.675937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:47.675980 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:47.675985 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:24:47.676108 (+ 123us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:24:47.676109 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:24:47.676143 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 14:24:47.676175 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:24:47.678112 8333 raft_consensus.cc:1261] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Refusing update from remote peer a20c036d76804f5785c139c24717477b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:47.678143 8193 raft_consensus.cc:1261] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Refusing update from remote peer a20c036d76804f5785c139c24717477b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:24:47.678323 8417 consensus_queue.cc:1035] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [LEADER]: Connected to new peer: Peer: permanent_uuid: "310d2dad493340c8a942642c21ed2467" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:47.678452 8414 consensus_queue.cc:1035] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e9be9ff83c14404aa332f9362c43686" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:24:47.688159 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:47.688277 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:24:47.688872 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bc73806f64a47a6b4048908fbc2f61c: Sending DeleteTablet for 3 replicas of tablet a0365de20e4642c398dc9a9c1300cff0
I20260620 14:24:47.689114 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet a0365de20e4642c398dc9a9c1300cff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41636
I20260620 14:24:47.689121 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet a0365de20e4642c398dc9a9c1300cff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51320
I20260620 14:24:47.689143 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet a0365de20e4642c398dc9a9c1300cff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:24:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52884
I20260620 14:24:47.689226 8430 tablet_replica.cc:330] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467: stopping tablet replica
I20260620 14:24:47.689226 8432 tablet_replica.cc:330] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b: stopping tablet replica
I20260620 14:24:47.689275 8430 raft_consensus.cc:2229] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:47.689280 8432 raft_consensus.cc:2229] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:24:47.689311 8430 raft_consensus.cc:2258] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:47.689333 8432 raft_consensus.cc:2258] T a0365de20e4642c398dc9a9c1300cff0 P a20c036d76804f5785c139c24717477b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:47.689420 8431 tablet_replica.cc:330] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686: stopping tablet replica
I20260620 14:24:47.689464 8431 raft_consensus.cc:2229] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:24:47.689491 8431 raft_consensus.cc:2258] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:24:47.689587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 7968
I20260620 14:24:47.691509 8431 ts_tablet_manager.cc:1905] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:47.693080 8431 ts_tablet_manager.cc:1918] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:47.693143 8431 log.cc:1196] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/a0365de20e4642c398dc9a9c1300cff0
I20260620 14:24:47.693377 8431 ts_tablet_manager.cc:1939] T a0365de20e4642c398dc9a9c1300cff0 P 1e9be9ff83c14404aa332f9362c43686: Deleting consensus metadata
I20260620 14:24:47.693717 7880 catalog_manager.cc:4841] TS 1e9be9ff83c14404aa332f9362c43686 (127.0.98.2:34095): tablet a0365de20e4642c398dc9a9c1300cff0 (table test-table [id=41f90a956d2b47e0bae59c5f2c045884]) successfully deleted
I20260620 14:24:47.694656 8430 ts_tablet_manager.cc:1905] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:24:47.695863 8430 ts_tablet_manager.cc:1918] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:24:47.695915 8430 log.cc:1196] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/a0365de20e4642c398dc9a9c1300cff0
I20260620 14:24:47.696142 8430 ts_tablet_manager.cc:1939] T a0365de20e4642c398dc9a9c1300cff0 P 310d2dad493340c8a942642c21ed2467: Deleting consensus metadata
I20260620 14:24:47.696457 7880 catalog_manager.cc:4841] TS 310d2dad493340c8a942642c21ed2467 (127.0.98.3:44429): tablet a0365de20e4642c398dc9a9c1300cff0 (table test-table [id=41f90a956d2b47e0bae59c5f2c045884]) successfully deleted
W20260620 14:24:47.699082 7881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39123 (error 108)
I20260620 14:24:47.699400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 8108
I20260620 14:24:47.705336 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:47.699452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39123 (local address 127.0.0.1:41668)
0620 14:24:47.705228 (+ 5776us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:47.705293 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5752,"thread_start_us":62,"threads_started":1}
I20260620 14:24:47.705438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 8248
W20260620 14:24:47.705503 7881 catalog_manager.cc:4584] TS a20c036d76804f5785c139c24717477b (127.0.98.1:39123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a0365de20e4642c398dc9a9c1300cff0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39123: connect: Connection refused (error 111)
I20260620 14:24:47.710431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 7863
2026-06-20T14:24:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2667 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:47 dist-test-slave-hfc3 krb5kdc[8494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:47 dist-test-slave-hfc3 krb5kdc[8494](info): set up 2 sockets
Jun 20 14:24:47 dist-test-slave-hfc3 krb5kdc[8494](info): commencing operation
krb5kdc: starting...
W20260620 14:24:48.747110 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:48 dist-test-slave-hfc3 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:48.868857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44161
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:48.948079 8507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:48.948287 8507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:48.948308 8507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:48.949445 8507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:48.949484 8507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:48.949497 8507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:48.949536 8507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:48.949563 8507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:48.951107 8507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:48.951357 8507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:48.951586 8507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:48.956429 8512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:48.956480 8507 server_base.cc:1034] running on GCE node
W20260620 14:24:48.956449 8515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:48.956429 8513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:48.956926 8507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:48.957252 8507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:48.958411 8507 hybrid_clock.cc:648] HybridClock initialized: now 1781965488958393 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T14:24:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1254 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:24:49 dist-test-slave-hfc3 krb5kdc[8520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:24:49 dist-test-slave-hfc3 krb5kdc[8520](info): set up 2 sockets
Jun 20 14:24:49 dist-test-slave-hfc3 krb5kdc[8520](info): commencing operation
krb5kdc: starting...
W20260620 14:24:50.015012 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:24:50 dist-test-slave-hfc3 krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:24:50.131029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34473
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:50.213296 8533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:50.213510 8533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:50.213533 8533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:50.214721 8533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:50.214764 8533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:50.214778 8533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:50.214792 8533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:50.214805 8533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:50.216377 8533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:50.216624 8533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:50.216874 8533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:50.221702 8538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:50.221760 8541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:50.221697 8539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:50.222030 8533 server_base.cc:1034] running on GCE node
I20260620 14:24:50.222262 8533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:50.222574 8533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:50.223748 8533 hybrid_clock.cc:648] HybridClock initialized: now 1781965490223741 us; error 45 us; skew 500 ppm
Jun 20 14:24:50 dist-test-slave-hfc3 krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:50.226256 8533 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:50.226738 8533 webserver.cc:460] Webserver started at http://127.0.98.62:46729/ using document root <none> and password file <none>
I20260620 14:24:50.226984 8533 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:50.227052 8533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:50.227201 8533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:50.228260 8533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "efab2ccf04054455af80a2210d93adcc"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "839f545c8f6379912dabeb6346503ddc"
server_key_iv: "404bd142f78cd6fa6bb69232dde5a40f"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.228614 8533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "efab2ccf04054455af80a2210d93adcc"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "839f545c8f6379912dabeb6346503ddc"
server_key_iv: "404bd142f78cd6fa6bb69232dde5a40f"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.229935 8533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:24:50.230683 8548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.230856 8533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.230957 8533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "efab2ccf04054455af80a2210d93adcc"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "839f545c8f6379912dabeb6346503ddc"
server_key_iv: "404bd142f78cd6fa6bb69232dde5a40f"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.231040 8533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:50.254565 8533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:50.256801 8533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:50.256983 8533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:50.261214 8533 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34473
I20260620 14:24:50.261251 8600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34473 every 8 connection(s)
I20260620 14:24:50.261806 8533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:50.262607 8601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:24:50.265038 8601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc: Bootstrap starting.
I20260620 14:24:50.265662 8601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc: Neither blocks nor log segments found. Creating new log.
I20260620 14:24:50.265974 8601 log.cc:824] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:50.266089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 8533
I20260620 14:24:50.266215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:24:50.266467 392 external_mini_cluster.cc:1428] Setting key a9b57e76a54953bb0781c1496c7a17f6
I20260620 14:24:50.266784 8601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc: No bootstrap required, opened a new log
I20260620 14:24:50.269212 8601 raft_consensus.cc:348] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab2ccf04054455af80a2210d93adcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34473 } }
I20260620 14:24:50.269378 8601 raft_consensus.cc:374] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:24:50.269429 8601 raft_consensus.cc:729] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efab2ccf04054455af80a2210d93adcc, State: Initialized, Role: FOLLOWER
I20260620 14:24:50.269591 8601 consensus_queue.cc:260] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab2ccf04054455af80a2210d93adcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34473 } }
I20260620 14:24:50.269690 8601 raft_consensus.cc:388] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:50.269734 8601 raft_consensus.cc:482] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:50.269793 8601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:24:50.270431 8601 raft_consensus.cc:504] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab2ccf04054455af80a2210d93adcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34473 } }
I20260620 14:24:50.270550 8601 leader_election.cc:302] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efab2ccf04054455af80a2210d93adcc; no voters:
I20260620 14:24:50.270766 8601 leader_election.cc:288] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:24:50.270787 8606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:24:50.270905 8606 raft_consensus.cc:686] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 1 LEADER]: Becoming Leader. State: Replica: efab2ccf04054455af80a2210d93adcc, State: Running, Role: LEADER
I20260620 14:24:50.270984 8606 consensus_queue.cc:237] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab2ccf04054455af80a2210d93adcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34473 } }
I20260620 14:24:50.271032 8601 sys_catalog.cc:564] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:50.271324 8606 sys_catalog.cc:455] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efab2ccf04054455af80a2210d93adcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab2ccf04054455af80a2210d93adcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34473 } } }
I20260620 14:24:50.271404 8606 sys_catalog.cc:458] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:50.271539 8606 sys_catalog.cc:455] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader efab2ccf04054455af80a2210d93adcc. Latest consensus state: current_term: 1 leader_uuid: "efab2ccf04054455af80a2210d93adcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab2ccf04054455af80a2210d93adcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34473 } } }
I20260620 14:24:50.271612 8606 sys_catalog.cc:458] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [sys.catalog]: This master's current role is: LEADER
W20260620 14:24:50.272960 8622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:24:50.273022 8622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:24:50.273125 8611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:50.273380 8611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:24:50 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:50.274906 8611 catalog_manager.cc:1347] Generated new cluster ID: 3c2b9547ca904aa19fac22d465802a4a
I20260620 14:24:50.274962 8611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:50.278803 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:50.267068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52542 (local address 127.0.98.62:34473)
0620 14:24:50.267272 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:50.267282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:50.267323 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:24:50.268072 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:50.268079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:50.268131 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:50.268308 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:50.268965 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.270071 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.271133 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.271760 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.271889 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:50.274975 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:50.274979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:50.274979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:50.274991 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:50.277031 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:50.277472 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:50.277474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:50.277475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:50.277577 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:50.277798 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:50.277799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:50.277799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:50.278095 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:50.278238 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:50.278424 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 14:24:50.278580 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20260620 14:24:50.279788 8611 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:24:50.280237 8611 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:50.283571 8611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc: Generated new TSK 0
I20260620 14:24:50.283737 8611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:24:50.311174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34473
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:50.392468 8627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:50.392702 8627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:50.392741 8627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:50.393877 8627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:50.393944 8627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:50.393988 8627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:50.395529 8627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:50.395795 8627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:50.396034 8627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:50.401335 8633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:50.401362 8635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:50.401362 8632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:50.401686 8627 server_base.cc:1034] running on GCE node
I20260620 14:24:50.401925 8627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:50.402170 8627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:50.403335 8627 hybrid_clock.cc:648] HybridClock initialized: now 1781965490403334 us; error 46 us; skew 500 ppm
Jun 20 14:24:50 dist-test-slave-hfc3 krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:50.405907 8627 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:24:50.406302 8627 webserver.cc:460] Webserver started at http://127.0.98.1:38177/ using document root <none> and password file <none>
I20260620 14:24:50.406553 8627 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:50.406598 8627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:50.406687 8627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:50.407639 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "0230f3a31c834a8495444d2cd6823a6f"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "6c29bcd8c09632f0be0b7ed922d382f0"
server_key_iv: "2c556a579d29e189d0e53bad19dd58a7"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.407936 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0230f3a31c834a8495444d2cd6823a6f"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "6c29bcd8c09632f0be0b7ed922d382f0"
server_key_iv: "2c556a579d29e189d0e53bad19dd58a7"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.409137 8627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:50.409758 8642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.409910 8627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:24:50.409991 8627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "0230f3a31c834a8495444d2cd6823a6f"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "6c29bcd8c09632f0be0b7ed922d382f0"
server_key_iv: "2c556a579d29e189d0e53bad19dd58a7"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.410043 8627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:50.432418 8627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:50.434537 8627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:50.434653 8627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:50.434918 8627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:50.435238 8627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:50.435272 8627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.435318 8627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:50.435344 8627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.441039 8627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38613
I20260620 14:24:50.441126 8755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38613 every 8 connection(s)
I20260620 14:24:50.441434 8627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:50.446556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 8627
I20260620 14:24:50.446640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:24:50.446839 392 external_mini_cluster.cc:1428] Setting key 460396f2eabc18da942154f308f9a8da
Jun 20 14:24:50 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:50.451002 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:50.442164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51805 (local address 127.0.98.62:34473)
0620 14:24:50.442242 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:50.442244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:50.442424 (+ 180us) server_negotiation.cc:407] Connection header received
0620 14:24:50.443370 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:50.443372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:50.443382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:50.443430 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:50.444005 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.444505 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.445474 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.445659 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.445709 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:50.447918 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:50.447920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:50.447920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:50.447928 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:50.449466 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:50.449959 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:50.449960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:50.449960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:50.450011 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:50.450293 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:50.450294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:50.450294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:50.450459 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:50.450513 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:50.450872 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 14:24:50.450939 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:24:50.451303 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:50.442120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:51805)
0620 14:24:50.442299 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:50.442355 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:50.443280 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:50.443491 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:50.443495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:50.443742 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:50.443931 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:50.443935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.444586 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.444587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:50.445400 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:50.445402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.445731 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.445731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:50.445831 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:50.446282 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:50.446289 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:50.447828 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:50.449548 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:50.449555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:50.449557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:50.449882 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:50.450076 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:50.450077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:50.450077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:50.450221 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:50.450527 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:50.450534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:50.450733 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 14:24:50.450856 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:24:50.451032 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260620 14:24:50.451845 8756 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34473
I20260620 14:24:50.451954 8756 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:50.452159 8756 heartbeater.cc:511] Master 127.0.98.62:34473 requested a full tablet report, sending...
I20260620 14:24:50.452690 8565 ts_manager.cc:194] Registered new tserver with Master: 0230f3a31c834a8495444d2cd6823a6f (127.0.98.1:38613)
I20260620 14:24:50.453310 8565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51805
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:24:50.472532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34473
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:50.553211 8761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:50.553411 8761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:50.553434 8761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:50.554538 8761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:50.554574 8761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:50.554600 8761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:50.556020 8761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:50.556195 8761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:50.556365 8761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:50.561259 8767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:50.561299 8766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:50.561260 8769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:50.561573 8761 server_base.cc:1034] running on GCE node
I20260620 14:24:50.561797 8761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:50.562041 8761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:50.563179 8761 hybrid_clock.cc:648] HybridClock initialized: now 1781965490563166 us; error 28 us; skew 500 ppm
Jun 20 14:24:50 dist-test-slave-hfc3 krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:50.565675 8761 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:24:50.566111 8761 webserver.cc:460] Webserver started at http://127.0.98.2:46655/ using document root <none> and password file <none>
I20260620 14:24:50.566404 8761 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:50.566454 8761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:50.566617 8761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:50.567625 8761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "ca4a5eb25f2b4798a9365e3df9d57478"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "20514c6d6765bb832ce8b6e57a3c6f2b"
server_key_iv: "76197c551ba42b721784ed886dfe1304"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.567930 8761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ca4a5eb25f2b4798a9365e3df9d57478"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "20514c6d6765bb832ce8b6e57a3c6f2b"
server_key_iv: "76197c551ba42b721784ed886dfe1304"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.569185 8761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:24:50.569914 8776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.570101 8761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:50.570187 8761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "ca4a5eb25f2b4798a9365e3df9d57478"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "20514c6d6765bb832ce8b6e57a3c6f2b"
server_key_iv: "76197c551ba42b721784ed886dfe1304"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.570261 8761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:50.583913 8761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:50.585716 8761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:50.585924 8761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:50.586182 8761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:50.586534 8761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:50.586570 8761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.586592 8761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:50.586639 8761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.592434 8761 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45571
I20260620 14:24:50.592476 8889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45571 every 8 connection(s)
I20260620 14:24:50.592797 8761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:24:50.597456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 8761
I20260620 14:24:50.597575 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:24:50.597816 392 external_mini_cluster.cc:1428] Setting key 0a7b66474d4f91a906c29ccf50164501
Jun 20 14:24:50 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:50.602728 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:50.593514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45647 (local address 127.0.98.62:34473)
0620 14:24:50.593574 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:50.593576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:50.593760 (+ 184us) server_negotiation.cc:407] Connection header received
0620 14:24:50.594818 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:50.594820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:50.594832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:50.594891 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:50.595503 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.596012 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.597046 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.597258 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.597366 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:50.599730 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:50.599732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:50.599732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:50.599740 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:50.601227 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:50.601741 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:50.601742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:50.601742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:50.601801 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:50.602085 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:50.602085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:50.602086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:50.602187 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:50.602240 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:50.602579 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 14:24:50.602647 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:24:50.603035 8892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:50.593490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:45647)
0620 14:24:50.593643 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:50.593678 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:50.594699 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:50.594968 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:50.594971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:50.595223 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:50.595407 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:50.595411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.596121 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.596123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:50.596953 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:50.596955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.597376 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.597377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:50.597499 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:50.598152 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:50.598162 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:50.599636 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:50.601330 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:50.601336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:50.601338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:50.601646 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:50.601875 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:50.601876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:50.601876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:50.602020 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:50.602250 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:50.602255 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:50.602437 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 14:24:50.602559 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:24:50.602749 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260620 14:24:50.603547 8890 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34473
I20260620 14:24:50.603647 8890 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:50.603847 8890 heartbeater.cc:511] Master 127.0.98.62:34473 requested a full tablet report, sending...
I20260620 14:24:50.604220 8565 ts_manager.cc:194] Registered new tserver with Master: ca4a5eb25f2b4798a9365e3df9d57478 (127.0.98.2:45571)
I20260620 14:24:50.604553 8565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45647
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:24:50.623168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34473
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:50.703511 8895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:50.703743 8895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:50.703768 8895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:50.704962 8895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:50.705013 8895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:50.705071 8895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:50.706604 8895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:50.706849 8895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:50.707087 8895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:50.712154 8903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:50.712154 8900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:50.712152 8901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:50.712392 8895 server_base.cc:1034] running on GCE node
I20260620 14:24:50.712589 8895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:50.712889 8895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:50.714046 8895 hybrid_clock.cc:648] HybridClock initialized: now 1781965490714038 us; error 33 us; skew 500 ppm
Jun 20 14:24:50 dist-test-slave-hfc3 krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:50.716583 8895 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:24:50.717092 8895 webserver.cc:460] Webserver started at http://127.0.98.3:44161/ using document root <none> and password file <none>
I20260620 14:24:50.717332 8895 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:50.717379 8895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:50.717519 8895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:24:50.718668 8895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "c9597a70917d48d2a312fc9d9e915ad8"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "a796a1b4989c64617af8dadbcd072ae3"
server_key_iv: "f3d97e0424c4c94073115df5a764203d"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.719022 8895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c9597a70917d48d2a312fc9d9e915ad8"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "a796a1b4989c64617af8dadbcd072ae3"
server_key_iv: "f3d97e0424c4c94073115df5a764203d"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.720327 8895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:50.721025 8910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.721216 8895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:50.721302 8895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "c9597a70917d48d2a312fc9d9e915ad8"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "a796a1b4989c64617af8dadbcd072ae3"
server_key_iv: "f3d97e0424c4c94073115df5a764203d"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.721356 8895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:50.734899 8895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:50.736644 8895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:50.736749 8895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:50.736927 8895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:50.737218 8895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:50.737251 8895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.737272 8895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:50.737318 8895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.743018 8895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36095
I20260620 14:24:50.743094 9023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36095 every 8 connection(s)
I20260620 14:24:50.743391 8895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:50.747821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 8895
I20260620 14:24:50.747939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:24:50.748147 392 external_mini_cluster.cc:1428] Setting key 8dbc8b9eb2b64e4b50d2f0f1e72d00c9
Jun 20 14:24:50 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:24:50.752856 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:50.744131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57831 (local address 127.0.98.62:34473)
0620 14:24:50.744216 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:50.744219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:50.744407 (+ 188us) server_negotiation.cc:407] Connection header received
0620 14:24:50.745367 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:50.745369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:50.745381 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:50.745437 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:50.746041 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.746640 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.747641 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.747975 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.748047 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:50.750322 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:50.750323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:50.750323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:50.750369 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:50.751800 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:50.752129 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:50.752130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:50.752130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:50.752179 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:50.752378 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:50.752378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:50.752378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:50.752473 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:50.752521 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:50.752721 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 14:24:50.752794 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:24:50.753012 9026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:50.744099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:57831)
0620 14:24:50.744290 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:50.744324 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:50.745254 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:50.745513 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:50.745517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:50.745755 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:50.745944 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:50.745948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.746732 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.746733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:50.747535 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:50.747538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:50.748074 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:50.748075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:50.748200 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:50.748792 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:50.748801 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:50.750244 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:24:50.751866 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:50.751871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:50.751872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:50.752068 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:50.752233 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:24:50.752234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:24:50.752234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:24:50.752326 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:24:50.752526 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:24:50.752529 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:24:50.752634 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 14:24:50.752711 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:24:50.752825 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260620 14:24:50.753458 9024 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34473
I20260620 14:24:50.753551 9024 heartbeater.cc:464] Registering TS with master...
I20260620 14:24:50.753683 9024 heartbeater.cc:511] Master 127.0.98.62:34473 requested a full tablet report, sending...
I20260620 14:24:50.754005 8565 ts_manager.cc:194] Registered new tserver with Master: c9597a70917d48d2a312fc9d9e915ad8 (127.0.98.3:36095)
I20260620 14:24:50.754385 8565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57831
I20260620 14:24:50.759990 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:24:50.826550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 8627
I20260620 14:24:50.830807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 8761
I20260620 14:24:50.835078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 8895
I20260620 14:24:50.839198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 8533
I20260620 14:24:50.844357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34473
--webserver_interface=127.0.98.62
--webserver_port=46729
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:50.924561 9033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:50.924824 9033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:50.924865 9033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:50.926000 9033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:24:50.926069 9033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:50.926091 9033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:50.926112 9033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:24:50.926142 9033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:24:50.927613 9033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:50.927886 9033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:50.928108 9033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:50.932878 9041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:50.932870 9039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:50.932868 9038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:50.933091 9033 server_base.cc:1034] running on GCE node
I20260620 14:24:50.933313 9033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:50.933600 9033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:50.934759 9033 hybrid_clock.cc:648] HybridClock initialized: now 1781965490934751 us; error 39 us; skew 500 ppm
Jun 20 14:24:50 dist-test-slave-hfc3 krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:50.937144 9033 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:24:50.937570 9033 webserver.cc:460] Webserver started at http://127.0.98.62:46729/ using document root <none> and password file <none>
I20260620 14:24:50.937801 9033 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:50.937881 9033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:50.939277 9033 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:24:50.939942 9048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:50.940109 9033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:50.940209 9033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "efab2ccf04054455af80a2210d93adcc"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "839f545c8f6379912dabeb6346503ddc"
server_key_iv: "404bd142f78cd6fa6bb69232dde5a40f"
server_key_version: "encryptionkey@0"
I20260620 14:24:50.940549 9033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:50.948966 9033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:50.951057 9033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:50.951215 9033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:50.955051 9033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34473
I20260620 14:24:50.955081 9100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34473 every 8 connection(s)
I20260620 14:24:50.955595 9033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:24:50.956817 9101 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:24:50.957474 9101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc: Bootstrap starting.
I20260620 14:24:50.958778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 9033
I20260620 14:24:50.959074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38613
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34473
--webserver_port=38177
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:24:50.966178 9101 log.cc:824] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:24:50.968063 9101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:24:50.968392 9101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc: Bootstrap complete.
I20260620 14:24:50.970716 9101 raft_consensus.cc:348] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab2ccf04054455af80a2210d93adcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34473 } }
I20260620 14:24:50.970971 9101 raft_consensus.cc:729] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efab2ccf04054455af80a2210d93adcc, State: Initialized, Role: FOLLOWER
I20260620 14:24:50.971156 9101 consensus_queue.cc:260] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab2ccf04054455af80a2210d93adcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34473 } }
I20260620 14:24:50.971235 9101 raft_consensus.cc:388] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:24:50.971319 9101 raft_consensus.cc:482] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:24:50.971371 9101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:24:50.972353 9101 raft_consensus.cc:504] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab2ccf04054455af80a2210d93adcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34473 } }
I20260620 14:24:50.972474 9101 leader_election.cc:302] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efab2ccf04054455af80a2210d93adcc; no voters:
I20260620 14:24:50.972679 9101 leader_election.cc:288] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:24:50.972721 9105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:24:50.972851 9105 raft_consensus.cc:686] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [term 2 LEADER]: Becoming Leader. State: Replica: efab2ccf04054455af80a2210d93adcc, State: Running, Role: LEADER
I20260620 14:24:50.972936 9105 consensus_queue.cc:237] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab2ccf04054455af80a2210d93adcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34473 } }
I20260620 14:24:50.972971 9101 sys_catalog.cc:564] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:24:50.973169 9105 sys_catalog.cc:455] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader efab2ccf04054455af80a2210d93adcc. Latest consensus state: current_term: 2 leader_uuid: "efab2ccf04054455af80a2210d93adcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab2ccf04054455af80a2210d93adcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34473 } } }
I20260620 14:24:50.973235 9105 sys_catalog.cc:458] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:50.973330 9107 sys_catalog.cc:455] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "efab2ccf04054455af80a2210d93adcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab2ccf04054455af80a2210d93adcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34473 } } }
I20260620 14:24:50.973430 9109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:24:50.973459 9107 sys_catalog.cc:458] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc [sys.catalog]: This master's current role is: LEADER
I20260620 14:24:50.973985 9109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:24:50.974149 9109 catalog_manager.cc:1259] Loaded cluster ID: 3c2b9547ca904aa19fac22d465802a4a
I20260620 14:24:50.974191 9109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:24:50.975392 9109 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:24:50.975594 9109 catalog_manager.cc:5879] T 00000000000000000000000000000000 P efab2ccf04054455af80a2210d93adcc: Loaded TSK: 0
I20260620 14:24:50.975817 9109 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:24:51.054193 9103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:51.054478 9103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:51.054509 9103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:51.055682 9103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:51.055724 9103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:51.055750 9103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:24:51.057163 9103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38177
--tserver_master_addrs=127.0.98.62:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:51.057403 9103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:51.057650 9103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:51.062628 9128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:51.062628 9127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:51.062852 9103 server_base.cc:1034] running on GCE node
W20260620 14:24:51.062937 9130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:51.063143 9103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:51.063395 9103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:51.064558 9103 hybrid_clock.cc:648] HybridClock initialized: now 1781965491064543 us; error 31 us; skew 500 ppm
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965491, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:51.067395 9103 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:24:51.067956 9103 webserver.cc:460] Webserver started at http://127.0.98.1:38177/ using document root <none> and password file <none>
I20260620 14:24:51.068245 9103 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:51.068344 9103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:51.070113 9103 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:24:51.070817 9137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:51.070999 9103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:24:51.071105 9103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "0230f3a31c834a8495444d2cd6823a6f"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "6c29bcd8c09632f0be0b7ed922d382f0"
server_key_iv: "2c556a579d29e189d0e53bad19dd58a7"
server_key_version: "encryptionkey@0"
I20260620 14:24:51.071475 9103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:51.094430 9103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:51.096822 9103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:51.096972 9103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:51.097221 9103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:51.097532 9103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:51.097565 9103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:51.097608 9103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:51.097635 9103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:51.103987 9103 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38613
I20260620 14:24:51.104027 9250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38613 every 8 connection(s)
I20260620 14:24:51.104383 9103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:24:51.105196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 9103
I20260620 14:24:51.105535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45571
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34473
--webserver_port=46655
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:51.114733 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:51.105374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:35059)
0620 14:24:51.106482 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:51.106521 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:51.107041 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:51.107864 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:51.107868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:51.108642 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:51.108838 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.108843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.110013 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.110016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.110898 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.110900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.111520 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.111521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.111659 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.112223 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:51.112231 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:51.114512 (+ 2281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1062,"thread_start_us":119,"threads_started":1}
W20260620 14:24:51.114993 9251 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34473 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34473: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260620 14:24:51.114777 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:51.105419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35059 (local address 127.0.98.62:34473)
0620 14:24:51.107278 (+ 1859us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:51.107288 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:51.107333 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:24:51.107506 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:51.107513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:51.107573 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:51.107709 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:51.108968 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.109911 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.111004 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.111416 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.111546 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.114633 (+ 3087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35059: BlockingRecv error: recv got EOF from 127.0.98.1:35059 (error 108)
Metrics: {"server-negotiator.queue_time_us":1799,"thread_start_us":84,"threads_started":1}
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:51.119747 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:51.115213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:37633)
0620 14:24:51.115498 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:51.115508 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:51.115558 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:51.115705 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:51.115706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:51.115844 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:51.115891 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.115893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.116546 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.116547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.117065 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.117067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.117427 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.117428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.117501 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.117938 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:51.117943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:51.119657 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":71,"threads_started":1}
W20260620 14:24:51.119784 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:51.115416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37633 (local address 127.0.98.62:34473)
0620 14:24:51.115473 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:51.115475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:51.115544 (+ 69us) server_negotiation.cc:407] Connection header received
0620 14:24:51.115617 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:51.115618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:51.115631 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:51.115703 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:51.115977 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.116463 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.117166 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.117355 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.117431 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.119727 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37633: BlockingRecv error: recv got EOF from 127.0.98.1:37633 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:51.124457 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:51.120330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:47689)
0620 14:24:51.120419 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:51.120429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:51.120473 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:51.120599 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:51.120599 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:51.120724 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:51.120751 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.120752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.121363 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.121365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.121861 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.121862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.122226 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.122227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.122280 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.122749 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:51.122754 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:51.124371 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 14:24:51.124503 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:51.120381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47689 (local address 127.0.98.62:34473)
0620 14:24:51.120417 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:51.120419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:51.120459 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:24:51.120529 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:51.120531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:51.120540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:51.120596 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:51.120831 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.121278 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.121957 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.122142 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.122226 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.124441 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47689: BlockingRecv error: recv got EOF from 127.0.98.1:47689 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:24:51.124850 9251 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:24:51.194530 9255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:51.194739 9255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:51.194763 9255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:51.195864 9255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:51.195899 9255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:51.195924 9255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:24:51.197290 9255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46655
--tserver_master_addrs=127.0.98.62:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:51.197471 9255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:51.197641 9255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:51.202634 9264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:51.202685 9262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:51.202661 9261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:51.203150 9255 server_base.cc:1034] running on GCE node
I20260620 14:24:51.203435 9255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:51.203696 9255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:51.204895 9255 hybrid_clock.cc:648] HybridClock initialized: now 1781965491204871 us; error 30 us; skew 500 ppm
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965491, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:51.207477 9255 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:24:51.207978 9255 webserver.cc:460] Webserver started at http://127.0.98.2:46655/ using document root <none> and password file <none>
I20260620 14:24:51.208253 9255 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:51.208343 9255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:51.209717 9255 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:51.210402 9271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:51.210577 9255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:51.210678 9255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "ca4a5eb25f2b4798a9365e3df9d57478"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "20514c6d6765bb832ce8b6e57a3c6f2b"
server_key_iv: "76197c551ba42b721784ed886dfe1304"
server_key_version: "encryptionkey@0"
I20260620 14:24:51.211030 9255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:51.235570 9255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:51.237584 9255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:51.237754 9255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:51.237994 9255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:51.238382 9255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:51.238446 9255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:51.238492 9255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:51.238520 9255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:51.244709 9255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45571
I20260620 14:24:51.244757 9384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45571 every 8 connection(s)
I20260620 14:24:51.245093 9255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:51.252025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 9255
I20260620 14:24:51.252329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36095
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34473
--webserver_port=44161
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:24:51.252866 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:51.245842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49069 (local address 127.0.98.62:34473)
0620 14:24:51.245908 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:51.245910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:51.246121 (+ 211us) server_negotiation.cc:407] Connection header received
0620 14:24:51.246650 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:51.246651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:51.246667 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:51.246738 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:51.247719 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.248208 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.249283 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.249463 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.249520 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.252811 (+ 3291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49069: BlockingRecv error: recv got EOF from 127.0.98.2:49069 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:24:51.252916 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:51.245776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:49069)
0620 14:24:51.245985 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:51.246023 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:51.246562 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:51.246799 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:51.246803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:51.247411 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:51.247614 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.247619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.248308 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.248309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.249198 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.249200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.249540 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.249540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.249649 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.250232 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:51.250240 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:51.252719 (+ 2479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":66,"threads_started":1}
W20260620 14:24:51.253122 9385 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34473 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34473: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:51.257740 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:51.253329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:58061)
0620 14:24:51.253362 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:51.253373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:51.253438 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:51.253582 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:51.253583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:51.253711 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:51.253754 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.253755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.254415 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.254417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.254924 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.254926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.255282 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.255283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.255348 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.255833 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:51.255837 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:51.257636 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 14:24:51.257750 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:51.253392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58061 (local address 127.0.98.62:34473)
0620 14:24:51.253462 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:51.253464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:51.253473 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:24:51.253496 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:51.253497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:51.253506 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:51.253579 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:51.253844 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.254314 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.255018 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.255201 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.255288 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.257707 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58061: BlockingRecv error: recv got EOF from 127.0.98.2:58061 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:51.262965 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:51.258094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:56315)
0620 14:24:51.258151 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:51.258162 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:51.258205 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:51.258389 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:51.258390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:51.258503 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:51.258528 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.258529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.259630 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.259631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.260107 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.260108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.260733 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.260734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.260774 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.261277 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:51.261281 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:51.262920 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:24:51.263135 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:51.258167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56315 (local address 127.0.98.62:34473)
0620 14:24:51.258233 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:51.258236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:51.258254 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:24:51.258279 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:51.258280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:51.258292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:51.258384 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:51.258605 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.259524 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.260193 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.260640 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.260730 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.263042 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56315: BlockingRecv error: recv got EOF from 127.0.98.2:56315 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 14:24:51.263276 9385 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:24:51.339432 9388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:24:51.339637 9388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:24:51.339661 9388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:24:51.340854 9388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:24:51.340919 9388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:24:51.340950 9388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:24:51.342444 9388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44161
--tserver_master_addrs=127.0.98.62:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:24:51.342681 9388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:24:51.342931 9388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:24:51.348073 9396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:51.348090 9393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:24:51.348086 9394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:24:51.348330 9388 server_base.cc:1034] running on GCE node
I20260620 14:24:51.348531 9388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:24:51.348800 9388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:24:51.349967 9388 hybrid_clock.cc:648] HybridClock initialized: now 1781965491349954 us; error 48 us; skew 500 ppm
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965491, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:24:51.352670 9388 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:24:51.353051 9388 webserver.cc:460] Webserver started at http://127.0.98.3:44161/ using document root <none> and password file <none>
I20260620 14:24:51.353231 9388 fs_manager.cc:359] Metadata directory not provided
I20260620 14:24:51.353283 9388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:24:51.354661 9388 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:24:51.355262 9403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:51.355435 9388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:24:51.355525 9388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "c9597a70917d48d2a312fc9d9e915ad8"
format_stamp: "Formatted at 2026-06-20 14:24:50 on dist-test-slave-hfc3"
server_key: "a796a1b4989c64617af8dadbcd072ae3"
server_key_iv: "f3d97e0424c4c94073115df5a764203d"
server_key_version: "encryptionkey@0"
I20260620 14:24:51.355876 9388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:24:51.368428 9388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:24:51.370203 9388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:24:51.370384 9388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:24:51.370661 9388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:24:51.370971 9388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:24:51.371004 9388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:51.371047 9388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:24:51.371074 9388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:24:51.376984 9388 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36095
I20260620 14:24:51.377063 9516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36095 every 8 connection(s)
I20260620 14:24:51.377391 9388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:24:51.378129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 9388
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:24:51.385365 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:51.378157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43701 (local address 127.0.98.62:34473)
0620 14:24:51.378229 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:51.378232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:51.378488 (+ 256us) server_negotiation.cc:407] Connection header received
0620 14:24:51.379216 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:51.379216 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:51.379223 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:51.379256 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:51.380331 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.380793 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.382122 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.382295 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.382426 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.385306 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43701: BlockingRecv error: recv got EOF from 127.0.98.3:43701 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:24:51.385519 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:51.378154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:43701)
0620 14:24:51.378325 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:51.378390 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:51.379115 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:51.379343 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:51.379347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:51.380047 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:51.380235 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.380240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.380880 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.380881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.382031 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.382038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.382435 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.382436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.382594 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.383084 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:51.383092 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:51.385240 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
W20260620 14:24:51.385854 9517 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34473 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34473: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260620 14:24:51.386595 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:51.378706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52556 (local address 127.0.98.62:34473)
0620 14:24:51.378815 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:51.378816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:51.378826 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:24:51.378889 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:51.378889 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:51.378900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:51.378953 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:51.379778 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.380300 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.381180 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.381363 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.381415 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.382912 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:24:51.382917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:24:51.382917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:24:51.382929 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:24:51.384868 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:51.385235 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:51.385238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:51.385238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:51.385334 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:24:51.385535 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:24:51.385536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:24:51.385536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:24:51.385831 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:24:51.385969 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:24:51.386325 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 14:24:51.386461 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:51.390229 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:51.386076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:50089)
0620 14:24:51.386128 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:51.386139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:51.386177 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:51.386326 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:51.386326 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:51.386498 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:51.386554 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.386556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.387342 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.387343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.387849 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.387851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.388214 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.388215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.388283 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.388613 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:51.388618 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:51.390184 (+ 1566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:24:51.390378 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:51.386124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50089 (local address 127.0.98.62:34473)
0620 14:24:51.386157 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:51.386159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:51.386167 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:51.386248 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:51.386248 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:51.386258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:51.386311 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:51.386638 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.387276 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.387946 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.388147 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.388217 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.390279 (+ 2062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50089: BlockingRecv error: recv got EOF from 127.0.98.3:50089 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:51 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:51.394282 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:51.390535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:59025)
0620 14:24:51.390656 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:51.390666 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:51.390715 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:51.390808 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:51.390808 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:51.390916 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:51.390955 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.390956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.391519 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.391520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.391999 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:51.392001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.392293 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.392294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:51.392330 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.392649 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:51.392653 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:51.394225 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20260620 14:24:51.394377 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:51.390608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59025 (local address 127.0.98.62:34473)
0620 14:24:51.390663 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:51.390665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:51.390702 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:24:51.390753 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:51.390754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:51.390761 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:51.390804 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:51.391009 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.391435 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.392070 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:51.392234 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:51.392281 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:51.394299 (+ 2018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59025: BlockingRecv error: recv got EOF from 127.0.98.3:59025 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 14:24:51.394497 9517 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:24:52 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:52 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:52.134819 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:52.125321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:57477)
0620 14:24:52.125454 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:52.125466 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:52.125543 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:52.125722 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:52.125723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:52.125944 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:52.126031 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:52.126034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:52.126689 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:52.126690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:52.127226 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:52.127228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:52.127567 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:52.127568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:52.127615 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:52.132685 (+ 5070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:52.132690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:52.134704 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
W20260620 14:24:52.134865 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:52.125400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57477 (local address 127.0.98.62:34473)
0620 14:24:52.125541 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:52.125544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:52.125552 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:52.125622 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:52.125623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:52.125635 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:52.125703 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:52.126108 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:52.126622 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:52.127303 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:52.127503 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:52.127555 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:52.134793 (+ 7238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57477: BlockingRecv error: recv got EOF from 127.0.98.1:57477 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
Jun 20 14:24:52 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:52 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:52.268700 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:52.263718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:37483)
0620 14:24:52.263942 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:52.263956 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:52.264070 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:52.264226 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:52.264227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:52.264456 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:52.264519 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:52.264521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:52.265274 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:52.265275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:52.265831 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:52.265833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:52.266211 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:52.266212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:52.266270 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:52.266811 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:52.266817 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:52.268593 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":126,"threads_started":1}
W20260620 14:24:52.268743 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:52.263859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37483 (local address 127.0.98.62:34473)
0620 14:24:52.263942 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:52.263944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:52.264050 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:24:52.264128 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:52.264129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:52.264143 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:52.264221 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:52.264608 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:52.265209 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:52.265920 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:52.266123 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:52.266210 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:52.268678 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37483: BlockingRecv error: recv got EOF from 127.0.98.2:37483 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 14:24:52 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:52 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:52.400041 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:52.394937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:39869)
0620 14:24:52.395186 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:52.395197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:52.395260 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:52.395469 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:52.395470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:52.395648 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:52.395715 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:52.395717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:52.396539 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:52.396540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:52.397108 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:52.397109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:52.397462 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:52.397463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:52.397520 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:52.398037 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:52.398042 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:52.399973 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":67,"threads_started":1}
W20260620 14:24:52.400285 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:52.394997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39869 (local address 127.0.98.62:34473)
0620 14:24:52.395179 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:52.395181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:52.395244 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:24:52.395341 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:52.395342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:52.395358 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:52.395434 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:52.395799 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:52.396447 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:52.397189 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:52.397373 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:52.397428 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:52.400195 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39869: BlockingRecv error: recv got EOF from 127.0.98.3:39869 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
Jun 20 14:24:53 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:53 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:53.139750 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:53.135390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:49493)
0620 14:24:53.135519 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:53.135532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:53.135585 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:53.135741 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:53.135742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:53.135896 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:53.135945 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:53.135946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.136572 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.136573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:53.137107 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:53.137109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.137432 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.137432 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:53.137471 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:53.137915 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:53.137919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:53.139691 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
W20260620 14:24:53.139886 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:53.135477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49493 (local address 127.0.98.62:34473)
0620 14:24:53.135605 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:53.135607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:53.135615 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:53.135641 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:53.135641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:53.135661 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:53.135723 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:53.136014 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.136505 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.137191 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.137371 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.137420 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:53.139777 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49493: BlockingRecv error: recv got EOF from 127.0.98.1:49493 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
Jun 20 14:24:53 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:53 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:53.274143 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:53.269181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:58681)
0620 14:24:53.269352 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:53.269364 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:53.269438 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:53.269599 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:53.269600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:53.269779 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:53.269826 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:53.269828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.270567 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.270568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:53.271157 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:53.271159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.271522 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.271523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:53.271578 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:53.272088 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:53.272099 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:53.274017 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
W20260620 14:24:53.274256 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:53.269272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58681 (local address 127.0.98.62:34473)
0620 14:24:53.269360 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:53.269367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:53.269417 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:24:53.269502 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:53.269503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:53.269514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:53.269600 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:53.269905 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.270488 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.271243 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.271449 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.271525 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:53.274180 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58681: BlockingRecv error: recv got EOF from 127.0.98.2:58681 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 14:24:53 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:53 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:53.405580 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:53.400727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:38895)
0620 14:24:53.400883 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:53.400894 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:53.400953 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:53.401188 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:53.401189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:53.401345 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:53.401391 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:53.401392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.402071 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.402072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:53.402680 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:53.402682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.403051 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.403052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:53.403128 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:53.403626 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:53.403631 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:53.405443 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
W20260620 14:24:53.405599 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:53.400868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38895 (local address 127.0.98.62:34473)
0620 14:24:53.401030 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:53.401033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:53.401048 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:24:53.401077 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:53.401078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:53.401091 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:53.401148 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:53.401483 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.401985 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.402766 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:53.402961 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:53.403051 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:53.405522 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38895: BlockingRecv error: recv got EOF from 127.0.98.3:38895 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Jun 20 14:24:54 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:54 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:54.145113 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:54.140205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:47355)
0620 14:24:54.140339 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:54.140350 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:54.140417 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:54.140570 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:54.140570 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:54.140731 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:54.140780 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:54.140781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.141441 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.141442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:54.142040 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:54.142041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.142446 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.142447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:54.142491 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:54.142919 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:54.142924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:54.145016 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
W20260620 14:24:54.145176 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:54.140238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47355 (local address 127.0.98.62:34473)
0620 14:24:54.140365 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:54.140367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:54.140408 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:24:54.140473 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:54.140474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:54.140484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:54.140553 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:54.140849 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.141359 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.142121 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.142327 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.142450 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:54.145103 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47355: BlockingRecv error: recv got EOF from 127.0.98.1:47355 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
Jun 20 14:24:54 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:54 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:54.279588 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:54.274722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:52893)
0620 14:24:54.274916 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:54.274931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:54.275011 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:54.275187 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:54.275188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:54.275366 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:54.275413 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:54.275415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.276096 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.276097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:54.276642 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:54.276643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.277019 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.277020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:54.277080 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:54.277521 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:54.277525 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:54.279522 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
W20260620 14:24:54.279716 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:54.274803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52893 (local address 127.0.98.62:34473)
0620 14:24:54.274920 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:54.274923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:54.275004 (+ 81us) server_negotiation.cc:407] Connection header received
0620 14:24:54.275077 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:54.275078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:54.275092 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:54.275164 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:54.275491 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.276007 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.276738 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.276938 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.277010 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:54.279631 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52893: BlockingRecv error: recv got EOF from 127.0.98.2:52893 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 14:24:54 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:54 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:54.411062 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:54.406245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:59773)
0620 14:24:54.406399 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:54.406410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:54.406478 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:54.406617 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:54.406617 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:54.406796 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:54.406845 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:54.406846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.407486 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.407486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:54.408006 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:54.408008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.408354 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.408354 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:54.408391 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:54.408901 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:54.408907 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:54.410994 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
W20260620 14:24:54.411237 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:54.406375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59773 (local address 127.0.98.62:34473)
0620 14:24:54.406425 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:54.406427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:54.406463 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:24:54.406541 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:54.406541 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:54.406555 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:54.406598 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:54.406919 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.407418 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.408087 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:54.408289 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:54.408344 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:54.411166 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59773: BlockingRecv error: recv got EOF from 127.0.98.3:59773 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 14:24:55 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:55 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:55.150847 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:55.145614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:41273)
0620 14:24:55.145767 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:55.145786 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:55.145840 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:55.145983 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:55.145984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:55.146153 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:55.146202 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:55.146204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.146958 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.146960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:55.147529 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:55.147532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.147978 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.147979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:55.148050 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:55.148591 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:55.148595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:55.150748 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
W20260620 14:24:55.150909 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:55.145691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41273 (local address 127.0.98.62:34473)
0620 14:24:55.145826 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:55.145828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:55.145841 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:24:55.145886 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:55.145887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:55.145898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:55.145965 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:55.146268 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.146844 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.147617 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.147889 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.147945 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:55.150835 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41273: BlockingRecv error: recv got EOF from 127.0.98.1:41273 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
Jun 20 14:24:55 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:55 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:55.284418 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:55.280039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:54883)
0620 14:24:55.280177 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:55.280193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:55.280249 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:55.280380 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:55.280380 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:55.280542 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:55.280590 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:55.280592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.281225 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.281226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:55.281754 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:55.281755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.282103 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.282104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:55.282144 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:55.282691 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:55.282696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:55.284362 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
W20260620 14:24:55.284533 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:55.280105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54883 (local address 127.0.98.62:34473)
0620 14:24:55.280178 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:55.280180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:55.280267 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:24:55.280307 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:55.280307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:55.280319 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:55.280362 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:55.280660 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.281152 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.281841 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.282037 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.282092 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:55.284457 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54883: BlockingRecv error: recv got EOF from 127.0.98.2:54883 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 14:24:55 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:55 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:55.416437 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:55.411583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:49717)
0620 14:24:55.411791 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:55.411810 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:55.411894 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:55.412073 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:55.412074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:55.412261 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:55.412313 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:55.412315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.413003 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.413004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:55.413646 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:55.413650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.414040 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.414041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:55.414098 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:55.414614 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:55.414621 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:55.416377 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
W20260620 14:24:55.416584 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:55.411687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49717 (local address 127.0.98.62:34473)
0620 14:24:55.411772 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:55.411774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:55.411873 (+ 99us) server_negotiation.cc:407] Connection header received
0620 14:24:55.411973 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:55.411974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:55.411990 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:55.412067 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:55.412394 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.412915 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.413754 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:55.413965 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:55.414042 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:55.416502 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49717: BlockingRecv error: recv got EOF from 127.0.98.3:49717 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 14:24:56 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:56 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:56.156599 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:56.151326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:42637)
0620 14:24:56.151461 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:56.151471 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:56.151520 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:56.151699 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:56.151700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:56.151857 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:56.151906 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:56.151908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:56.152885 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:56.152886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:56.153550 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:56.153553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:56.153919 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:56.153920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:56.153957 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:56.154530 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:56.154534 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:56.156542 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
W20260620 14:24:56.156845 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:56.151447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42637 (local address 127.0.98.62:34473)
0620 14:24:56.151574 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:56.151575 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:56.151583 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:56.151606 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:56.151607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:56.151617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:56.151687 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:56.152010 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:56.152793 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:56.153631 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:56.153850 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:56.153913 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:56.156778 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42637: BlockingRecv error: recv got EOF from 127.0.98.1:42637 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
Jun 20 14:24:56 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:56 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:56.289497 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:56.284977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:56543)
0620 14:24:56.285145 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:56.285163 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:56.285223 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:56.285378 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:56.285379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:56.285549 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:56.285597 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:56.285599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:56.286259 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:56.286260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:56.286855 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:56.286856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:56.287193 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:56.287194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:56.287232 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:56.287659 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:56.287663 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:56.289438 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
W20260620 14:24:56.289645 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:56.285112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56543 (local address 127.0.98.62:34473)
0620 14:24:56.285181 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:56.285183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:56.285209 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:24:56.285291 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:56.285291 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:56.285302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:56.285361 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:56.285661 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:56.286193 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:56.286938 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:56.287126 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:56.287183 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:56.289555 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56543: BlockingRecv error: recv got EOF from 127.0.98.2:56543 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 14:24:56 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:56 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:56.422016 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:56.416930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:46005)
0620 14:24:56.417162 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:56.417175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:56.417275 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:56.417504 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:56.417505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:56.417690 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:56.417737 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:56.417739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:56.418476 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:56.418477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:56.419043 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:56.419045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:56.419429 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:56.419430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:56.419490 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:56.420003 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:56.420009 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:56.421895 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
W20260620 14:24:56.422070 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:56.417054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46005 (local address 127.0.98.62:34473)
0620 14:24:56.417148 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:56.417151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:56.417253 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:24:56.417401 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:56.417402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:56.417417 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:56.417495 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:56.417821 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:56.418389 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:56.419141 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:56.419351 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:56.419430 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:56.421999 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46005: BlockingRecv error: recv got EOF from 127.0.98.3:46005 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 14:24:57 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:57 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:57.166507 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:57.157196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:38031)
0620 14:24:57.157341 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:57.157356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:57.157415 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:57.157548 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:57.157548 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:57.157711 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:57.157758 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:57.157759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.158454 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.158455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:57.159004 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:57.159006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.159378 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.159378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:57.159420 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:57.164510 (+ 5090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:57.164516 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:57.166434 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
W20260620 14:24:57.166659 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:57.157245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38031 (local address 127.0.98.62:34473)
0620 14:24:57.157373 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:57.157375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:57.157413 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:24:57.157458 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:57.157458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:57.157469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:57.157544 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:57.157819 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.158396 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.159125 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.159330 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.159378 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:57.166563 (+ 7185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38031: BlockingRecv error: recv got EOF from 127.0.98.1:38031 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
Jun 20 14:24:57 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:57 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:57.295104 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:57.290013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:44321)
0620 14:24:57.290229 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:57.290242 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:57.290339 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:57.290545 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:57.290546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:57.290743 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:57.290790 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:57.290792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.291485 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.291486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:57.292034 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:57.292035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.292456 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.292457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:57.292537 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:57.293080 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:57.293084 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:57.295034 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":145,"threads_started":1}
W20260620 14:24:57.295413 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:57.290120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44321 (local address 127.0.98.62:34473)
0620 14:24:57.290214 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:57.290216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:57.290321 (+ 105us) server_negotiation.cc:407] Connection header received
0620 14:24:57.290448 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:57.290450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:57.290464 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:57.290537 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:57.290879 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.291397 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.292129 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.292349 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.292421 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:57.295295 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44321: BlockingRecv error: recv got EOF from 127.0.98.2:44321 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 14:24:57 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:57 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:57.427634 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:57.422715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:44477)
0620 14:24:57.422891 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:57.422902 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:57.422952 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:57.423102 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:57.423103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:57.423267 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:57.423316 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:57.423318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.424009 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.424010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:57.424783 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:57.424785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.425127 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.425128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:57.425169 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:57.425633 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:57.425638 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:57.427575 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
W20260620 14:24:57.427788 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:57.422824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44477 (local address 127.0.98.62:34473)
0620 14:24:57.422932 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:57.422934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:57.422947 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:24:57.423004 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:57.423005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:57.423019 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:57.423081 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:57.423412 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.423922 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.424867 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:57.425051 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:57.425120 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:57.427702 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44477: BlockingRecv error: recv got EOF from 127.0.98.3:44477 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 14:24:58 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:58 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:58.171537 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.166948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:56541)
0620 14:24:58.167097 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.167107 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:58.167161 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:58.167288 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:58.167289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:58.167446 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:58.167493 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.167494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.168106 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.168106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.168634 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.168635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.168943 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.168944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.168982 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.169413 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:58.169418 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:58.171438 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
W20260620 14:24:58.171589 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.167023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56541 (local address 127.0.98.62:34473)
0620 14:24:58.167131 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.167133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.167145 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:24:58.167218 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.167219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.167229 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.167284 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.167546 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.168041 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.168700 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.168885 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.168934 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.171517 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56541: BlockingRecv error: recv got EOF from 127.0.98.1:56541 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":50,"threads_started":1}
Jun 20 14:24:58 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:58 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:58.300417 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.295611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:42845)
0620 14:24:58.295768 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.295785 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:58.295843 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:58.295978 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:58.295978 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:58.296140 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:58.296196 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.296197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.296847 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.296849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.297391 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.297392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.297711 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.297712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.297752 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.298180 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:58.298184 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:58.300359 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
W20260620 14:24:58.300518 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.295699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42845 (local address 127.0.98.62:34473)
0620 14:24:58.295789 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.295792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.295853 (+ 61us) server_negotiation.cc:407] Connection header received
0620 14:24:58.295883 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.295884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.295894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.295956 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.296262 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.296755 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.297468 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.297641 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.297698 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.300451 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42845: BlockingRecv error: recv got EOF from 127.0.98.2:42845 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 14:24:58 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:58 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:58.433051 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:58.428160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:37455)
0620 14:24:58.428330 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:58.428341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:58.428401 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:58.428554 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:58.428555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:58.428718 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:58.428767 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.428768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.429410 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.429411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.429964 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:58.429965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.430459 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.430460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:58.430531 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.431109 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:58.431114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:58.432927 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
W20260620 14:24:58.433082 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:58.428248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37455 (local address 127.0.98.62:34473)
0620 14:24:58.428330 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:58.428332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:58.428391 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:24:58.428447 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:58.428448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:58.428462 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:58.428519 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:58.428834 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.429331 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.430079 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:58.430333 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:58.430460 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:58.433019 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37455: BlockingRecv error: recv got EOF from 127.0.98.3:37455 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 14:24:59 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:59 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:59.176806 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:59.172027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:59783)
0620 14:24:59.172220 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:59.172231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:59.172291 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:59.172431 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:59.172432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:59.172644 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:59.172692 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:59.172694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.173353 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.173354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:59.173966 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:59.173968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.174323 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.174323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:59.174436 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:59.174973 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:59.174977 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:59.176735 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":115,"threads_started":1}
W20260620 14:24:59.176911 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:59.172159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59783 (local address 127.0.98.62:34473)
0620 14:24:59.172315 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:59.172317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:59.172325 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:59.172348 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:59.172348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:59.172359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:59.172417 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:59.172745 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.173261 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.174038 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.174253 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.174315 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:59.176836 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59783: BlockingRecv error: recv got EOF from 127.0.98.1:59783 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
Jun 20 14:24:59 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:59 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:59.305744 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:59.300966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:38897)
0620 14:24:59.301105 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:59.301122 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:59.301186 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:59.301367 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:59.301368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:59.301564 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:59.301610 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:59.301612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.302465 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.302466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:59.303000 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:59.303002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.303391 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.303392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:59.303446 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:59.303921 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:59.303925 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:59.305645 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
W20260620 14:24:59.305790 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:59.301157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38897 (local address 127.0.98.62:34473)
0620 14:24:59.301229 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:59.301231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:59.301239 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:24:59.301267 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:59.301268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:59.301293 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:59.301359 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:59.301694 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.302379 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.303100 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.303306 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.303391 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:59.305728 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38897: BlockingRecv error: recv got EOF from 127.0.98.2:38897 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 14:24:59 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:24:59 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:24:59.437832 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:24:59.433588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:40747)
0620 14:24:59.433756 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:24:59.433768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:24:59.433837 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:24:59.433952 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:24:59.433953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:24:59.434111 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:24:59.434160 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:59.434161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.434827 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.434828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:59.435345 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:24:59.435347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.435648 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.435648 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:24:59.435685 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:59.436126 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:24:59.436131 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:24:59.437773 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":110,"threads_started":1}
W20260620 14:24:59.437956 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:24:59.433690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40747 (local address 127.0.98.62:34473)
0620 14:24:59.433748 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:24:59.433749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:24:59.433836 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:24:59.433880 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:24:59.433881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:24:59.433893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:24:59.433947 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:24:59.434214 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.434757 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.435412 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:24:59.435588 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:24:59.435636 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:24:59.437870 (+ 2234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40747: BlockingRecv error: recv got EOF from 127.0.98.3:40747 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 14:25:00 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:00 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:00.182454 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:00.177305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:46567)
0620 14:25:00.177451 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:00.177465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:00.177544 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:00.177771 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:00.177772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:00.177998 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:00.178069 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:00.178071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.179029 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.179030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:00.179561 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:00.179563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.179907 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.179908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:00.179951 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:00.180488 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:00.180493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:00.182391 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
W20260620 14:25:00.182566 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:00.177436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46567 (local address 127.0.98.62:34473)
0620 14:25:00.177594 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:00.177596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:00.177606 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:25:00.177636 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:00.177637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:00.177662 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:00.177737 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:00.178159 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.178952 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.179636 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.179837 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.179890 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:00.182464 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46567: BlockingRecv error: recv got EOF from 127.0.98.1:46567 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
Jun 20 14:25:00 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:00 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:00.311010 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:00.306402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:38045)
0620 14:25:00.306543 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:00.306553 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:00.306611 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:00.306730 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:00.306731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:00.306903 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:00.306953 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:00.306955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.307585 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.307586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:00.308181 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:00.308183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.308538 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.308539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:00.308581 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:00.309032 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:00.309037 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:00.310952 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
W20260620 14:25:00.311146 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:00.306411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38045 (local address 127.0.98.62:34473)
0620 14:25:00.306465 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:00.306468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:00.306619 (+ 151us) server_negotiation.cc:407] Connection header received
0620 14:25:00.306662 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:00.306662 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:00.306674 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:00.306719 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:00.307016 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.307502 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.308285 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.308470 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.308523 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:00.311064 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38045: BlockingRecv error: recv got EOF from 127.0.98.2:38045 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 14:25:00 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:00 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:00.443017 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:00.438521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:48411)
0620 14:25:00.438656 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:00.438668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:00.438725 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:00.438838 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:00.438839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:00.439000 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:00.439047 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:00.439048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.439681 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.439681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:00.440210 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:00.440212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.440539 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.440540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:00.440578 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:00.441031 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:00.441036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:00.442955 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
W20260620 14:25:00.443161 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:00.438640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48411 (local address 127.0.98.62:34473)
0620 14:25:00.438722 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:00.438725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:00.438734 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:25:00.438773 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:00.438773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:00.438786 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:00.438833 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:00.439109 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.439611 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.440286 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:00.440473 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:00.440525 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:00.443075 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48411: BlockingRecv error: recv got EOF from 127.0.98.3:48411 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 14:25:01 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:01 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:01.187945 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.182928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:51295)
0620 14:25:01.183111 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.183124 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:01.183208 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:01.183347 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:01.183348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:01.183517 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:01.183568 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.183570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.184260 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.184261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.184796 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.184798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.185185 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.185185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.185229 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.185735 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:01.185739 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:01.187800 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":126,"threads_started":1}
W20260620 14:25:01.188228 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.183032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51295 (local address 127.0.98.62:34473)
0620 14:25:01.183148 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.183150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.183207 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:25:01.183252 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.183253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.183265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.183332 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.183662 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.184178 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.184908 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.185114 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.185178 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.188160 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51295: BlockingRecv error: recv got EOF from 127.0.98.1:51295 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
Jun 20 14:25:01 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:01 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:01.317288 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.311584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:44357)
0620 14:25:01.311773 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.311785 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:01.311855 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:01.312021 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:01.312022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:01.312265 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:01.312327 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.312329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.313176 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.313178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.313913 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.313916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.314452 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.314453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.314500 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.315076 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:01.315081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:01.317182 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
W20260620 14:25:01.317351 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.311709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44357 (local address 127.0.98.62:34473)
0620 14:25:01.311762 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.311764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.311848 (+ 84us) server_negotiation.cc:407] Connection header received
0620 14:25:01.311917 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.311918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.311931 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.311998 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.312448 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.313064 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.314019 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.314237 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.314367 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.317281 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44357: BlockingRecv error: recv got EOF from 127.0.98.2:44357 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 14:25:01 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:01 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:01.448778 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:01.443612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:52229)
0620 14:25:01.443820 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:01.443877 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:01.443978 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:01.444165 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:01.444166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:01.444356 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:01.444405 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.444407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.445184 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.445185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.445714 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:01.445716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.446110 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.446110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:01.446153 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.446783 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:01.446788 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:01.448711 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
W20260620 14:25:01.448859 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:01.443749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52229 (local address 127.0.98.62:34473)
0620 14:25:01.443822 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:01.443824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:01.443966 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:25:01.444056 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:01.444057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:01.444074 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:01.444137 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:01.444513 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.445090 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.445802 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:01.446036 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:01.446099 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:01.448808 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52229: BlockingRecv error: recv got EOF from 127.0.98.3:52229 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 14:25:02 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:02 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:02.196245 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:02.188470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:43707)
0620 14:25:02.188628 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:02.188643 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:02.188707 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:02.188837 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:02.188838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:02.189003 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:02.189051 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:02.189052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.189688 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.189688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:02.190209 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:02.190210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.190548 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.190549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:02.190591 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:02.194083 (+ 3492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:02.194088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:02.196148 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
W20260620 14:25:02.196309 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:02.188533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43707 (local address 127.0.98.62:34473)
0620 14:25:02.188687 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:02.188689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:02.188698 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:25:02.188751 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:02.188752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:02.188761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:02.188819 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:02.189116 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.189621 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.190279 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.190487 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.190540 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:02.196233 (+ 5693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43707: BlockingRecv error: recv got EOF from 127.0.98.1:43707 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
Jun 20 14:25:02 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:02 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:02.322384 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:02.317888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:41159)
0620 14:25:02.318032 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:02.318043 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:02.318104 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:02.318235 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:02.318235 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:02.318456 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:02.318505 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:02.318506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.319141 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.319142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:02.319657 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:02.319658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.320081 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.320082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:02.320120 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:02.320572 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:02.320576 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:02.322281 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
W20260620 14:25:02.322521 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:02.317941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41159 (local address 127.0.98.62:34473)
0620 14:25:02.318000 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:02.318003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:02.318110 (+ 107us) server_negotiation.cc:407] Connection header received
0620 14:25:02.318158 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:02.318158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:02.318169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:02.318221 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:02.318574 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.319074 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.319760 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.320009 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.320076 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:02.322427 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41159: BlockingRecv error: recv got EOF from 127.0.98.2:41159 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 14:25:02 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:02 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:02.454092 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:02.449394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:33147)
0620 14:25:02.449603 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:02.449614 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:02.449701 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:02.449851 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:02.449852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:02.450032 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:02.450079 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:02.450081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.450818 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.450819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:02.451356 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:02.451358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.451716 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.451717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:02.451772 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:02.452243 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:02.452248 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:02.454006 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":117,"threads_started":1}
W20260620 14:25:02.454185 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:02.449490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33147 (local address 127.0.98.62:34473)
0620 14:25:02.449570 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:02.449572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:02.449683 (+ 111us) server_negotiation.cc:407] Connection header received
0620 14:25:02.449756 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:02.449757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:02.449773 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:02.449844 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:02.450158 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.450733 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.451453 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:02.451644 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:02.451725 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:02.454119 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33147: BlockingRecv error: recv got EOF from 127.0.98.3:33147 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 14:25:03 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:03 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:03.201424 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:03.196769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:46261)
0620 14:25:03.196927 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:03.196939 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:03.196994 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:03.197128 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:03.197154 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:03.197317 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:03.197368 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:03.197369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:03.198004 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:03.198005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:03.198615 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:03.198617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:03.198965 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:03.198966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:03.199003 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:03.199477 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:03.199482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:03.201331 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
W20260620 14:25:03.201496 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:03.196874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46261 (local address 127.0.98.62:34473)
0620 14:25:03.197005 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:03.197012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:03.197021 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:25:03.197042 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:03.197042 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:03.197053 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:03.197110 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:03.197437 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:03.197937 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:03.198695 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:03.198901 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:03.198957 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:03.201424 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46261: BlockingRecv error: recv got EOF from 127.0.98.1:46261 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
Jun 20 14:25:03 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:03 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:03.327699 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:03.322990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:42605)
0620 14:25:03.323177 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:03.323190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:03.323252 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:03.323394 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:03.323395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:03.323566 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:03.323615 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:03.323616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:03.324294 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:03.324295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:03.324852 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:03.324854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:03.325170 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:03.325171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:03.325219 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:03.325694 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:03.325699 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:03.327631 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":119,"threads_started":1}
W20260620 14:25:03.327829 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:03.323111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42605 (local address 127.0.98.62:34473)
0620 14:25:03.323182 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:03.323185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:03.323240 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:25:03.323323 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:03.323323 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:03.323334 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:03.323377 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:03.323707 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:03.324205 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:03.324928 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:03.325109 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:03.325158 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:03.327742 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42605: BlockingRecv error: recv got EOF from 127.0.98.2:42605 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 14:25:03 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:03 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:03.459482 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:03.454568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:44043)
0620 14:25:03.454735 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:03.454747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:03.454804 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:03.454969 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:03.454970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:03.455141 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:03.455189 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:03.455191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:03.455822 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:03.455822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:03.456334 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:03.456335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:03.456703 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:03.456704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:03.456755 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:03.457233 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:03.457238 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:03.459371 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":59,"threads_started":1}
W20260620 14:25:03.459537 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:03.454732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44043 (local address 127.0.98.62:34473)
0620 14:25:03.454799 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:03.454801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:03.454810 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:25:03.454859 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:03.454860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:03.454871 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:03.454920 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:03.455260 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:03.455754 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:03.456409 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:03.456607 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:03.456663 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:03.459485 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44043: BlockingRecv error: recv got EOF from 127.0.98.3:44043 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 14:25:04 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:04 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:04.206449 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:04.201967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:52323)
0620 14:25:04.202101 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:04.202113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:04.202173 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:04.202296 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:04.202297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:04.202514 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:04.202562 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:04.202564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:04.203199 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:04.203200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:04.203725 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:04.203727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:04.204059 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:04.204060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:04.204097 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:04.204519 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:04.204523 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:04.206321 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
W20260620 14:25:04.206497 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:04.202019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52323 (local address 127.0.98.62:34473)
0620 14:25:04.202148 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:04.202150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:04.202161 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:25:04.202231 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:04.202232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:04.202243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:04.202291 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:04.202629 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:04.203130 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:04.203803 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:04.203996 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:04.204045 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:04.206427 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52323: BlockingRecv error: recv got EOF from 127.0.98.1:52323 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
Jun 20 14:25:04 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:04 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:04.333182 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:04.328317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:33567)
0620 14:25:04.328606 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:04.328622 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:04.328680 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:04.328825 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:04.328825 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:04.328989 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:04.329036 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:04.329038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:04.329678 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:04.329679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:04.330223 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:04.330225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:04.330584 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:04.330585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:04.330625 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:04.331059 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:04.331064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:04.333115 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":192,"threads_started":1}
W20260620 14:25:04.333310 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:04.328549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33567 (local address 127.0.98.62:34473)
0620 14:25:04.328658 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:04.328659 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:04.328671 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:25:04.328744 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:04.328745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:04.328755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:04.328798 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:04.329103 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:04.329597 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:04.330307 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:04.330521 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:04.330577 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:04.333205 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33567: BlockingRecv error: recv got EOF from 127.0.98.2:33567 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 14:25:04 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:04 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:04.465189 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:04.460047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:39721)
0620 14:25:04.460251 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:04.460265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:04.460396 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:04.460554 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:04.460555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:04.460748 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:04.460797 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:04.460799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:04.461539 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:04.461540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:04.462078 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:04.462080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:04.462521 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:04.462522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:04.462586 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:04.463154 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:04.463159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:04.465094 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":111,"threads_started":1}
W20260620 14:25:04.465246 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:04.460165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39721 (local address 127.0.98.62:34473)
0620 14:25:04.460252 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:04.460254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:04.460377 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:25:04.460456 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:04.460457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:04.460479 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:04.460547 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:04.460883 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:04.461446 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:04.462170 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:04.462431 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:04.462512 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:04.465178 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39721: BlockingRecv error: recv got EOF from 127.0.98.3:39721 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 14:25:05 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:05 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:05.211417 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:05.206974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:40853)
0620 14:25:05.207100 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:05.207110 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:05.207159 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:05.207306 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:05.207306 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:05.207471 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:05.207518 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:05.207520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:05.208152 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:05.208153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:05.208700 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:05.208702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:05.209038 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:05.209038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:05.209078 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:05.209616 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:05.209621 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:05.211358 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
W20260620 14:25:05.211556 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:05.207046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40853 (local address 127.0.98.62:34473)
0620 14:25:05.207183 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:05.207185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:05.207193 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:25:05.207214 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:05.207215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:05.207224 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:05.207284 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:05.207584 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:05.208085 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:05.208775 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:05.208978 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:05.209028 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:05.211449 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40853: BlockingRecv error: recv got EOF from 127.0.98.1:40853 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
Jun 20 14:25:05 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:05 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:05.338248 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:05.333702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:42947)
0620 14:25:05.333836 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:05.333850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:05.333913 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:05.334046 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:05.334047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:05.334211 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:05.334259 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:05.334260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:05.334983 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:05.334984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:05.335548 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:05.335550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:05.335863 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:05.335864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:05.335906 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:05.336350 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:05.336354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:05.338151 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
W20260620 14:25:05.338297 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:05.333752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42947 (local address 127.0.98.62:34473)
0620 14:25:05.333837 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:05.333839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:05.333916 (+ 77us) server_negotiation.cc:407] Connection header received
0620 14:25:05.333979 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:05.333979 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:05.333990 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:05.334040 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:05.334426 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:05.334910 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:05.335622 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:05.335801 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:05.335850 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:05.338234 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42947: BlockingRecv error: recv got EOF from 127.0.98.2:42947 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 14:25:05 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:05 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:05.470088 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:05.465711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.3:35863)
0620 14:25:05.465894 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:05.465903 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:05.465961 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:05.466094 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:05.466094 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:05.466257 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:05.466304 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:05.466306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:05.466982 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:05.466983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:05.467515 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:05.467517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:05.467827 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:05.467827 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:05.467873 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:05.468316 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:05.468322 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:05.470018 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":105,"threads_started":1}
W20260620 14:25:05.470207 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:05.465762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35863 (local address 127.0.98.62:34473)
0620 14:25:05.465834 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:05.465836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:05.465961 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:25:05.466017 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:05.466018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:05.466030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:05.466070 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:05.466428 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:05.466913 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:05.467587 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:05.467767 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:05.467816 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:05.470109 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35863: BlockingRecv error: recv got EOF from 127.0.98.3:35863 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:25:06 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:06 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:06.216846 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:06.211948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.1:50071)
0620 14:25:06.212130 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:06.212141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:06.212197 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:06.212340 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:06.212340 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:06.212501 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:06.212548 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:06.212549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.213182 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.213183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:06.213790 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:06.213792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.214117 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.214118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:06.214156 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:06.214733 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:06.214738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:06.216786 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
W20260620 14:25:06.216958 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:06.212085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50071 (local address 127.0.98.62:34473)
0620 14:25:06.212214 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:06.212216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:06.212224 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:25:06.212246 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:06.212247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:06.212257 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:06.212321 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:06.212617 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.213114 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.213856 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.214053 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.214107 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:06.216879 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50071: BlockingRecv error: recv got EOF from 127.0.98.1:50071 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
Jun 20 14:25:06 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:25:06 dist-test-slave-hfc3 krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:25:06.343545 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:06.338882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34473 (local address 127.0.98.2:35599)
0620 14:25:06.339055 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:06.339066 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:06.339130 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:06.339262 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:06.339262 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:06.339423 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:06.339469 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:06.339505 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.340200 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.340201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:06.340749 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:06.340750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.341072 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.341073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:06.341111 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:06.341558 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:06.341562 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:06.343394 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
W20260620 14:25:06.343545 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:06.339018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35599 (local address 127.0.98.62:34473)
0620 14:25:06.339067 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:06.339069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:06.339122 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:25:06.339188 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:06.339189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:06.339200 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:06.339241 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:06.339613 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.340116 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.340825 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.341009 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.341064 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:06.343489 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35599: BlockingRecv error: recv got EOF from 127.0.98.2:35599 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:25:06.388876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 9103
I20260620 14:25:06.393564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 9255
I20260620 14:25:06.397785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 9388
I20260620 14:25:06.401978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 9033
2026-06-20T14:25:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17435 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T14:25:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:25:06Z Disabled control of system clock
I20260620 14:25:06.430636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42255
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:25:06.516130 9584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:06.516347 9584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:06.516368 9584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:06.517570 9584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:06.517618 9584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:06.517632 9584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:06.517644 9584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:06.517656 9584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:06.519063 9584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:25:06.519340 9584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:06.519586 9584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:06.527012 9590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:06.527057 9589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:06.527558 9584 server_base.cc:1034] running on GCE node
W20260620 14:25:06.527014 9592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:06.527976 9584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:06.528411 9584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:06.529681 9584 hybrid_clock.cc:648] HybridClock initialized: now 1781965506529595 us; error 80 us; skew 500 ppm
I20260620 14:25:06.532001 9584 webserver.cc:460] Webserver started at http://127.0.98.62:42295/ using document root <none> and password file <none>
I20260620 14:25:06.532626 9584 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:06.532677 9584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:06.533030 9584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:06.534996 9584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "aaa2876fddcb4e9ba08e258662fd98c4"
format_stamp: "Formatted at 2026-06-20 14:25:06 on dist-test-slave-hfc3"
server_key: "0009d8f25b1b9026c83b3c1a87ac8079"
server_key_iv: "4d70e7babc82922fca39ca135e2ba6f0"
server_key_version: "encryptionkey@0"
I20260620 14:25:06.535372 9584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "aaa2876fddcb4e9ba08e258662fd98c4"
format_stamp: "Formatted at 2026-06-20 14:25:06 on dist-test-slave-hfc3"
server_key: "0009d8f25b1b9026c83b3c1a87ac8079"
server_key_iv: "4d70e7babc82922fca39ca135e2ba6f0"
server_key_version: "encryptionkey@0"
I20260620 14:25:06.537456 9584 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:25:06.539042 9598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:06.539283 9584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:25:06.539395 9584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "aaa2876fddcb4e9ba08e258662fd98c4"
format_stamp: "Formatted at 2026-06-20 14:25:06 on dist-test-slave-hfc3"
server_key: "0009d8f25b1b9026c83b3c1a87ac8079"
server_key_iv: "4d70e7babc82922fca39ca135e2ba6f0"
server_key_version: "encryptionkey@0"
I20260620 14:25:06.539548 9584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:06.550544 9584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:06.550899 9584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:06.551039 9584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:06.555732 9584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42255
I20260620 14:25:06.555821 9650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42255 every 8 connection(s)
I20260620 14:25:06.556422 9584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:25:06.557165 9651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:06.560099 9651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4: Bootstrap starting.
I20260620 14:25:06.560737 9651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:06.560993 9651 log.cc:824] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:06.561707 9651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4: No bootstrap required, opened a new log
I20260620 14:25:06.563398 9651 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa2876fddcb4e9ba08e258662fd98c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42255 } }
I20260620 14:25:06.563555 9651 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:06.563580 9651 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa2876fddcb4e9ba08e258662fd98c4, State: Initialized, Role: FOLLOWER
I20260620 14:25:06.563694 9651 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa2876fddcb4e9ba08e258662fd98c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42255 } }
I20260620 14:25:06.563776 9651 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:25:06.563812 9651 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:25:06.563887 9651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:06.564481 9651 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa2876fddcb4e9ba08e258662fd98c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42255 } }
I20260620 14:25:06.564625 9651 leader_election.cc:302] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaa2876fddcb4e9ba08e258662fd98c4; no voters:
I20260620 14:25:06.564860 9651 leader_election.cc:288] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:25:06.564963 9654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:06.565119 9654 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [term 1 LEADER]: Becoming Leader. State: Replica: aaa2876fddcb4e9ba08e258662fd98c4, State: Running, Role: LEADER
I20260620 14:25:06.565140 9651 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:06.565227 9654 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa2876fddcb4e9ba08e258662fd98c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42255 } }
I20260620 14:25:06.565642 9656 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaa2876fddcb4e9ba08e258662fd98c4. Latest consensus state: current_term: 1 leader_uuid: "aaa2876fddcb4e9ba08e258662fd98c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa2876fddcb4e9ba08e258662fd98c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42255 } } }
I20260620 14:25:06.565903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 9584
I20260620 14:25:06.566143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:25:06.566136 9656 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:06.566406 9655 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaa2876fddcb4e9ba08e258662fd98c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa2876fddcb4e9ba08e258662fd98c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42255 } } }
I20260620 14:25:06.566442 392 external_mini_cluster.cc:1428] Setting key 2a23f2d87131ba0ce2111630ad86aa53
I20260620 14:25:06.566479 9655 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4 [sys.catalog]: This master's current role is: LEADER
W20260620 14:25:06.568996 9672 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:25:06.569183 9672 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:25:06.569396 9661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:25:06.569796 9661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:25:06.572324 9661 catalog_manager.cc:1347] Generated new cluster ID: d68b1a10967d4a44a88f6885c3adbabb
I20260620 14:25:06.572383 9661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:25:06.575626 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:06.567232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.0.98.62:42255)
0620 14:25:06.567664 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:06.567675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:06.567755 (+ 80us) server_negotiation.cc:407] Connection header received
0620 14:25:06.568048 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:06.568057 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:06.568176 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:06.568448 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:06.569256 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.572031 (+ 2775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.573238 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.574001 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.574135 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:06.574883 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:06.574887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:06.574888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:25:06.575115 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:06.575161 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:25:06.575177 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:06.575248 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:06.575316 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:25:06.575399 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":168,"threads_started":1}
I20260620 14:25:06.581656 9661 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:25:06.582233 9661 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:25:06.588208 9661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaa2876fddcb4e9ba08e258662fd98c4: Generated new TSK 0
I20260620 14:25:06.588390 9661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:25:06.596525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42255
--builtin_ntp_servers=127.0.98.20:37777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:25:06.868880 9675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:06.869518 9675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:06.869599 9675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:06.873600 9675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:06.873752 9675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:06.873819 9675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:25:06.878294 9675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:25:06.879269 9675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:06.880229 9675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:06.892951 9681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:06.893106 9680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:06.892927 9683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:06.894074 9675 server_base.cc:1034] running on GCE node
I20260620 14:25:06.894327 9675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:06.894980 9675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:06.896164 9675 hybrid_clock.cc:648] HybridClock initialized: now 1781965506896154 us; error 63 us; skew 500 ppm
I20260620 14:25:06.898309 9675 webserver.cc:460] Webserver started at http://127.0.98.1:41057/ using document root <none> and password file <none>
I20260620 14:25:06.898847 9675 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:06.898926 9675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:06.899106 9675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:06.901064 9675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "fa3cf227cae04eb4901f807b4d0aa361"
format_stamp: "Formatted at 2026-06-20 14:25:06 on dist-test-slave-hfc3"
server_key: "275934104b94eb1aa5876a1245f1b9aa"
server_key_iv: "54ce59fc21a39cdc30257fdbb6734be1"
server_key_version: "encryptionkey@0"
I20260620 14:25:06.901669 9675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fa3cf227cae04eb4901f807b4d0aa361"
format_stamp: "Formatted at 2026-06-20 14:25:06 on dist-test-slave-hfc3"
server_key: "275934104b94eb1aa5876a1245f1b9aa"
server_key_iv: "54ce59fc21a39cdc30257fdbb6734be1"
server_key_version: "encryptionkey@0"
I20260620 14:25:06.904822 9675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 14:25:06.907536 9689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:06.908149 9675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:25:06.908481 9675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "fa3cf227cae04eb4901f807b4d0aa361"
format_stamp: "Formatted at 2026-06-20 14:25:06 on dist-test-slave-hfc3"
server_key: "275934104b94eb1aa5876a1245f1b9aa"
server_key_iv: "54ce59fc21a39cdc30257fdbb6734be1"
server_key_version: "encryptionkey@0"
I20260620 14:25:06.908661 9675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:06.952447 9675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:06.953600 9675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:06.954166 9675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:06.955073 9675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:06.956274 9675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:06.956390 9675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:06.956704 9675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:06.956897 9675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:06.977556 9675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37911
I20260620 14:25:06.977658 9802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37911 every 8 connection(s)
I20260620 14:25:06.978608 9675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:25:06.986402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 9675
I20260620 14:25:06.986899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:25:06.988155 392 external_mini_cluster.cc:1428] Setting key 0d731e3a61bec1308fad40386fdb9380
I20260620 14:25:06.991226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42255
--builtin_ntp_servers=127.0.98.20:37777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:25:06.995674 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:06.980305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36361 (local address 127.0.98.62:42255)
0620 14:25:06.980418 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:06.980421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:06.981024 (+ 603us) server_negotiation.cc:407] Connection header received
0620 14:25:06.982377 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:06.982378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:06.982433 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:06.982537 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:06.984531 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.986518 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.991581 (+ 5063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.992277 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.992482 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:06.994904 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:06.994905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:06.994906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:25:06.994914 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:06.994965 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:25:06.994971 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:06.995055 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:06.995293 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 14:25:06.995401 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:25:06.996033 9805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:06.980318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42255 (local address 127.0.98.1:36361)
0620 14:25:06.980784 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:06.980856 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:06.982169 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:06.982651 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:06.982654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:06.982672 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:06.984190 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:06.984195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.986761 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.986762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:06.991251 (+ 4489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:06.991270 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:06.992637 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:06.992648 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:06.993209 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:06.994531 (+ 1322us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:25:06.994531 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:06.994619 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:25:06.994700 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:06.995080 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:06.995085 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:06.995088 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:25:06.995270 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 14:25:06.995597 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":193,"threads_started":1}
I20260620 14:25:06.997172 9803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42255
I20260620 14:25:06.997279 9803 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:06.997900 9803 heartbeater.cc:511] Master 127.0.98.62:42255 requested a full tablet report, sending...
I20260620 14:25:06.999183 9615 ts_manager.cc:194] Registered new tserver with Master: fa3cf227cae04eb4901f807b4d0aa361 (127.0.98.1:37911)
I20260620 14:25:07.001504 9615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36361
W20260620 14:25:07.278913 9806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:07.279423 9806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:07.279466 9806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:07.282729 9806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:07.282899 9806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:07.282940 9806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:25:07.287369 9806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:25:07.287925 9806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:07.288605 9806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:07.301506 9811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:07.301431 9812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:07.301441 9814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:07.302096 9806 server_base.cc:1034] running on GCE node
I20260620 14:25:07.302748 9806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:07.303337 9806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:07.304659 9806 hybrid_clock.cc:648] HybridClock initialized: now 1781965507304585 us; error 105 us; skew 500 ppm
I20260620 14:25:07.310410 9806 webserver.cc:460] Webserver started at http://127.0.98.2:41275/ using document root <none> and password file <none>
I20260620 14:25:07.310698 9806 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:07.310772 9806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:07.310931 9806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:07.312057 9806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "c0b69b8ff3e74dc5b19c3611aca54267"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "e3d63673c44655177bc4b61d54f7b4c8"
server_key_iv: "58a366dd623e14c67075f3585eacc404"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.312434 9806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c0b69b8ff3e74dc5b19c3611aca54267"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "e3d63673c44655177bc4b61d54f7b4c8"
server_key_iv: "58a366dd623e14c67075f3585eacc404"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.313827 9806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:25:07.314626 9820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.314796 9806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:07.314899 9806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "c0b69b8ff3e74dc5b19c3611aca54267"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "e3d63673c44655177bc4b61d54f7b4c8"
server_key_iv: "58a366dd623e14c67075f3585eacc404"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.314972 9806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:07.323479 9806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:07.323776 9806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:07.323931 9806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:07.324163 9806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:07.324504 9806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:07.324559 9806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.324594 9806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:07.324620 9806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.331025 9806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35549
I20260620 14:25:07.331097 9933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35549 every 8 connection(s)
I20260620 14:25:07.331426 9806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:25:07.336498 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:07.332193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44679 (local address 127.0.98.62:42255)
0620 14:25:07.332264 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:07.332266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:07.332532 (+ 266us) server_negotiation.cc:407] Connection header received
0620 14:25:07.333010 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:07.333011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:07.333023 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:07.333077 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:07.333769 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:07.334293 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:07.335317 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:07.335504 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:07.335555 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:07.336269 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:07.336270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:07.336270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:25:07.336274 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:07.336294 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:25:07.336298 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:07.336347 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:07.336408 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:25:07.336432 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:25:07.336709 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:07.332182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42255 (local address 127.0.98.2:44679)
0620 14:25:07.332356 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:07.332389 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:07.332911 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:07.333182 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:07.333185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:07.333194 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:07.333671 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:07.333675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:07.334452 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:07.334453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:07.335245 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:07.335247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:07.335574 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:07.335575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:07.335693 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:07.336147 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:25:07.336147 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:07.336173 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:25:07.336194 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:07.336347 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:07.336350 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:07.336352 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:25:07.336398 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:25:07.336469 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":90,"threads_started":1}
I20260620 14:25:07.337208 9934 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42255
I20260620 14:25:07.337312 9934 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:07.337489 9934 heartbeater.cc:511] Master 127.0.98.62:42255 requested a full tablet report, sending...
I20260620 14:25:07.337980 9615 ts_manager.cc:194] Registered new tserver with Master: c0b69b8ff3e74dc5b19c3611aca54267 (127.0.98.2:35549)
I20260620 14:25:07.338383 9615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44679
I20260620 14:25:07.338829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 9806
I20260620 14:25:07.338963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:25:07.339215 392 external_mini_cluster.cc:1428] Setting key c9fc1c59ee6c7f3d51ee9c377edd9ee2
I20260620 14:25:07.340163 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42255
--builtin_ntp_servers=127.0.98.20:37777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:25:07.423220 9937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:07.423434 9937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:07.423455 9937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:07.424641 9937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:07.424685 9937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:07.424710 9937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:25:07.426074 9937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:25:07.426357 9937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:07.426659 9937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:07.431764 9945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:07.431746 9943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:07.431945 9937 server_base.cc:1034] running on GCE node
W20260620 14:25:07.431746 9942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:07.432219 9937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:07.432430 9937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:07.433586 9937 hybrid_clock.cc:648] HybridClock initialized: now 1781965507433566 us; error 34 us; skew 500 ppm
I20260620 14:25:07.435014 9937 webserver.cc:460] Webserver started at http://127.0.98.3:41409/ using document root <none> and password file <none>
I20260620 14:25:07.435307 9937 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:07.435359 9937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:07.435452 9937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:07.436523 9937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "98e3195e0b024db19a8f610382d6beb3"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "cfc337abb538abe4c8e1eb3c547ace86"
server_key_iv: "ad6968b8f49979d755ec6468313fa7be"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.436846 9937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "98e3195e0b024db19a8f610382d6beb3"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "cfc337abb538abe4c8e1eb3c547ace86"
server_key_iv: "ad6968b8f49979d755ec6468313fa7be"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.438191 9937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:25:07.439002 9951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.439185 9937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:25:07.439286 9937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "98e3195e0b024db19a8f610382d6beb3"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "cfc337abb538abe4c8e1eb3c547ace86"
server_key_iv: "ad6968b8f49979d755ec6468313fa7be"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.439364 9937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:07.459522 9937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:07.459883 9937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:07.460062 9937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:07.460336 9937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:07.460721 9937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:07.460755 9937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.460798 9937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:07.460827 9937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.467767 9937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38561
I20260620 14:25:07.467852 10064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38561 every 8 connection(s)
I20260620 14:25:07.468205 9937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:25:07.473615 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:07.468967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47415 (local address 127.0.98.62:42255)
0620 14:25:07.469015 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:07.469017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:07.469292 (+ 275us) server_negotiation.cc:407] Connection header received
0620 14:25:07.469867 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:07.469869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:07.469880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:07.469957 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:07.470718 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:07.471235 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:07.472264 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:07.472465 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:07.472528 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:07.473375 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:07.473376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:07.473376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:25:07.473380 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:07.473393 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:25:07.473398 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:07.473446 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:07.473524 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:25:07.473553 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:25:07.473888 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:07.468962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42255 (local address 127.0.98.3:47415)
0620 14:25:07.469154 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:07.469189 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:07.469749 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:07.470015 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:07.470019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:07.470027 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:07.470622 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:07.470628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:07.471315 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:07.471316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:07.472176 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:07.472178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:07.472578 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:07.472578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:07.472701 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:07.473238 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:25:07.473238 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:07.473267 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:25:07.473289 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:07.473449 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:07.473451 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:07.473453 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:25:07.473506 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:25:07.473617 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260620 14:25:07.474448 10065 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42255
I20260620 14:25:07.474555 10065 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:07.474706 10065 heartbeater.cc:511] Master 127.0.98.62:42255 requested a full tablet report, sending...
I20260620 14:25:07.475108 9615 ts_manager.cc:194] Registered new tserver with Master: 98e3195e0b024db19a8f610382d6beb3 (127.0.98.3:38561)
I20260620 14:25:07.475517 9615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47415
I20260620 14:25:07.475937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 9937
I20260620 14:25:07.476059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:25:07.476269 392 external_mini_cluster.cc:1428] Setting key e5e91d819f1281cee2cbc1167e50e4ac
I20260620 14:25:07.477195 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:25:07.479087 9666 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:07.478591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.0.98.62:42255)
0620 14:25:07.478668 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:07.478670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:07.478781 (+ 111us) server_negotiation.cc:407] Connection header received
0620 14:25:07.478818 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:07.478818 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:07.478829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:07.478867 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:07.478994 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33272: BlockingRecv error: recv got EOF from 127.0.0.1:33272 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:25:07.479120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 9675
I20260620 14:25:07.483873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 9806
I20260620 14:25:07.488654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 9937
I20260620 14:25:07.493046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 9584
2026-06-20T14:25:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1090 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T14:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:25:07Z Disabled control of system clock
I20260620 14:25:07.520567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38145
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:25:07.607950 10076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:07.608160 10076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:07.608183 10076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:07.609251 10076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:07.609284 10076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:07.609297 10076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:07.609309 10076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:07.609323 10076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:07.611340 10076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38145
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:25:07.611629 10076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:07.612005 10076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:07.617964 10082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:07.618045 10081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:07.617964 10084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:07.618474 10076 server_base.cc:1034] running on GCE node
I20260620 14:25:07.618777 10076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:07.619129 10076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:07.620293 10076 hybrid_clock.cc:648] HybridClock initialized: now 1781965507620282 us; error 29 us; skew 500 ppm
I20260620 14:25:07.621905 10076 webserver.cc:460] Webserver started at http://127.0.98.62:40245/ using document root <none> and password file <none>
I20260620 14:25:07.622179 10076 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:07.622247 10076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:07.622493 10076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:07.623844 10076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "384146ef4cc047d3aa1adcb714d31490"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "338c1cbea94fbe128ffb7cfc928b44c1"
server_key_iv: "d6cb1be3c8b8d0aa8db250f17b4e6bb1"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.624269 10076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "384146ef4cc047d3aa1adcb714d31490"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "338c1cbea94fbe128ffb7cfc928b44c1"
server_key_iv: "d6cb1be3c8b8d0aa8db250f17b4e6bb1"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.626194 10076 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 14:25:07.627278 10090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.627471 10076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:25:07.627578 10076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "384146ef4cc047d3aa1adcb714d31490"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "338c1cbea94fbe128ffb7cfc928b44c1"
server_key_iv: "d6cb1be3c8b8d0aa8db250f17b4e6bb1"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.627667 10076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:07.631043 10076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:07.631330 10076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:07.631475 10076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:07.635757 10076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38145
I20260620 14:25:07.635849 10142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38145 every 8 connection(s)
I20260620 14:25:07.636206 10076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:25:07.637038 10143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:07.641116 10143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490: Bootstrap starting.
I20260620 14:25:07.641774 10143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:07.642092 10143 log.cc:824] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:07.642910 10143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490: No bootstrap required, opened a new log
I20260620 14:25:07.644596 10143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384146ef4cc047d3aa1adcb714d31490" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } }
I20260620 14:25:07.644743 10143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:07.644766 10143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384146ef4cc047d3aa1adcb714d31490, State: Initialized, Role: FOLLOWER
I20260620 14:25:07.644928 10143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384146ef4cc047d3aa1adcb714d31490" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } }
I20260620 14:25:07.645015 10143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:25:07.645043 10143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:25:07.644999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 10076
I20260620 14:25:07.645089 10143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:07.645114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:25:07.645306 392 external_mini_cluster.cc:1428] Setting key 19a6369483659438a5d156d6b8a16eeb
I20260620 14:25:07.645792 10143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384146ef4cc047d3aa1adcb714d31490" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } }
I20260620 14:25:07.645903 10143 leader_election.cc:302] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384146ef4cc047d3aa1adcb714d31490; no voters:
I20260620 14:25:07.646054 10143 leader_election.cc:288] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:25:07.646086 10148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:07.646205 10148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [term 1 LEADER]: Becoming Leader. State: Replica: 384146ef4cc047d3aa1adcb714d31490, State: Running, Role: LEADER
I20260620 14:25:07.646287 10148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384146ef4cc047d3aa1adcb714d31490" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } }
I20260620 14:25:07.646327 10143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:07.646651 10148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384146ef4cc047d3aa1adcb714d31490. Latest consensus state: current_term: 1 leader_uuid: "384146ef4cc047d3aa1adcb714d31490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384146ef4cc047d3aa1adcb714d31490" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } } }
I20260620 14:25:07.646740 10148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:07.646994 10153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:25:07.647140 10150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "384146ef4cc047d3aa1adcb714d31490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384146ef4cc047d3aa1adcb714d31490" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } } }
I20260620 14:25:07.647240 10150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:07.647596 10153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:25:07.648885 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:07.645840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.0.98.62:38145)
0620 14:25:07.646101 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:07.646111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:07.646153 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:25:07.646283 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:07.646290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:07.646368 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:07.646520 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:07.648224 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:07.648228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:07.648229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:25:07.648234 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:07.648317 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:25:07.648333 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:07.648394 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:07.648628 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 14:25:07.648698 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":101,"threads_started":1}
I20260620 14:25:07.650108 10153 catalog_manager.cc:1347] Generated new cluster ID: e019aea7d00a4331ac1276cf4a1010be
I20260620 14:25:07.650156 10153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:25:07.664527 10153 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:25:07.664767 10153 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:25:07.668597 10153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384146ef4cc047d3aa1adcb714d31490: Generated new TSK 0
I20260620 14:25:07.668813 10153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:25:07.673125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38145
--builtin_ntp_servers=127.0.98.20:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:25:07.768530 10167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:07.768779 10167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:07.768821 10167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:07.770036 10167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:07.770103 10167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:07.770152 10167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:25:07.771587 10167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:25:07.771868 10167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:07.772177 10167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:07.778115 10173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:07.778127 10172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:07.778437 10167 server_base.cc:1034] running on GCE node
W20260620 14:25:07.778448 10175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:07.778720 10167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:07.779017 10167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:07.780175 10167 hybrid_clock.cc:648] HybridClock initialized: now 1781965507780157 us; error 35 us; skew 500 ppm
I20260620 14:25:07.781483 10167 webserver.cc:460] Webserver started at http://127.0.98.1:40653/ using document root <none> and password file <none>
I20260620 14:25:07.781751 10167 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:07.781824 10167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:07.781980 10167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:07.783135 10167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "1da77055f11544199ef261ed39cf5c6b"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "5f6e1bb3a1468884bb547dd87413ed4b"
server_key_iv: "6d19efe6b48f8f09e16ba99fcf9e84ab"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.783502 10167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1da77055f11544199ef261ed39cf5c6b"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "5f6e1bb3a1468884bb547dd87413ed4b"
server_key_iv: "6d19efe6b48f8f09e16ba99fcf9e84ab"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.784960 10167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:25:07.785727 10181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.785899 10167 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:25:07.785988 10167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "1da77055f11544199ef261ed39cf5c6b"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "5f6e1bb3a1468884bb547dd87413ed4b"
server_key_iv: "6d19efe6b48f8f09e16ba99fcf9e84ab"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.786059 10167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:07.789794 10167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:07.790073 10167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:07.790206 10167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:07.790490 10167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:07.790850 10167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:07.790902 10167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.790939 10167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:07.790963 10167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.798609 10167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40771
I20260620 14:25:07.798650 10294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40771 every 8 connection(s)
I20260620 14:25:07.799037 10167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:25:07.802506 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:07.799811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53695 (local address 127.0.98.62:38145)
0620 14:25:07.799888 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:07.799891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:07.800170 (+ 279us) server_negotiation.cc:407] Connection header received
0620 14:25:07.800861 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:07.800862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:07.800868 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:07.800912 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:07.802238 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:07.802241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:07.802241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:25:07.802247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:07.802263 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:25:07.802267 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:07.802321 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:07.802416 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:25:07.802432 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:25:07.802866 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:07.799833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38145 (local address 127.0.98.1:53695)
0620 14:25:07.800029 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:07.800072 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:07.800751 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:07.801008 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:07.801013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:07.801026 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:07.802084 (+ 1058us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:25:07.802084 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:07.802112 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:25:07.802139 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:07.802317 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:07.802321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:07.802325 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:25:07.802426 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:25:07.802487 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260620 14:25:07.803346 10295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38145
I20260620 14:25:07.803432 10295 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:07.803591 10295 heartbeater.cc:511] Master 127.0.98.62:38145 requested a full tablet report, sending...
I20260620 14:25:07.804121 10107 ts_manager.cc:194] Registered new tserver with Master: 1da77055f11544199ef261ed39cf5c6b (127.0.98.1:40771)
I20260620 14:25:07.809082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 10167
I20260620 14:25:07.809195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:25:07.809420 392 external_mini_cluster.cc:1428] Setting key 754431998b6ca2ae917e57f25e39c761
I20260620 14:25:07.810420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38145
--builtin_ntp_servers=127.0.98.20:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:25:07.894129 10298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:07.894393 10298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:07.894431 10298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:07.895582 10298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:07.895618 10298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:07.895644 10298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:25:07.896924 10298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:25:07.897114 10298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:07.897357 10298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:07.902241 10304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:07.902304 10303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:07.902314 10298 server_base.cc:1034] running on GCE node
W20260620 14:25:07.902261 10306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:07.902765 10298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:07.903021 10298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:07.904191 10298 hybrid_clock.cc:648] HybridClock initialized: now 1781965507904197 us; error 46 us; skew 500 ppm
I20260620 14:25:07.905659 10298 webserver.cc:460] Webserver started at http://127.0.98.2:36721/ using document root <none> and password file <none>
I20260620 14:25:07.905939 10298 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:07.906010 10298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:07.906170 10298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:07.907287 10298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "7799278e81d04c14999bec969e9b1e5b"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "ad725d0543891c16166bc13042cbc666"
server_key_iv: "81910e463d295db8ca0c91f6e5931521"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.907651 10298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7799278e81d04c14999bec969e9b1e5b"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "ad725d0543891c16166bc13042cbc666"
server_key_iv: "81910e463d295db8ca0c91f6e5931521"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.908968 10298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:25:07.909699 10312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.909910 10298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:07.910007 10298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "7799278e81d04c14999bec969e9b1e5b"
format_stamp: "Formatted at 2026-06-20 14:25:07 on dist-test-slave-hfc3"
server_key: "ad725d0543891c16166bc13042cbc666"
server_key_iv: "81910e463d295db8ca0c91f6e5931521"
server_key_version: "encryptionkey@0"
I20260620 14:25:07.910095 10298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:07.913463 10298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:07.913736 10298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:07.913887 10298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:07.914090 10298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:07.914450 10298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:07.914502 10298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.914542 10298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:07.914570 10298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:07.920290 10298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42153
I20260620 14:25:07.920360 10425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42153 every 8 connection(s)
I20260620 14:25:07.920723 10298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:25:07.923763 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:07.921465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33589 (local address 127.0.98.62:38145)
0620 14:25:07.921537 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:07.921539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:07.921772 (+ 233us) server_negotiation.cc:407] Connection header received
0620 14:25:07.922280 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:07.922281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:07.922288 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:07.922367 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:07.923474 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:07.923476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:07.923476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:25:07.923481 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:07.923498 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:25:07.923502 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:07.923577 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:07.923669 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:25:07.923685 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:25:07.923982 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:07.921472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38145 (local address 127.0.98.2:33589)
0620 14:25:07.921650 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:07.921688 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:07.922167 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:07.922429 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:07.922433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:07.922442 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:07.923323 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:25:07.923323 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:07.923351 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:25:07.923378 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:07.923572 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:07.923577 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:07.923581 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:25:07.923707 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:25:07.923768 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260620 14:25:07.924417 10426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38145
I20260620 14:25:07.924512 10426 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:07.924669 10426 heartbeater.cc:511] Master 127.0.98.62:38145 requested a full tablet report, sending...
I20260620 14:25:07.925024 10107 ts_manager.cc:194] Registered new tserver with Master: 7799278e81d04c14999bec969e9b1e5b (127.0.98.2:42153)
I20260620 14:25:07.925376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 10298
I20260620 14:25:07.925478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:25:07.925685 392 external_mini_cluster.cc:1428] Setting key 8758772f69a3363c3c41eb1a68e1ec4c
I20260620 14:25:07.926585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38145
--builtin_ntp_servers=127.0.98.20:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:25:08.007263 10429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:08.007481 10429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:08.007517 10429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:08.008649 10429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:08.008715 10429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:08.008755 10429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:25:08.010080 10429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:25:08.010361 10429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:08.010644 10429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:08.015654 10434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:08.015681 10429 server_base.cc:1034] running on GCE node
W20260620 14:25:08.015654 10435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:08.015715 10437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:08.016225 10429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:08.016459 10429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:08.017623 10429 hybrid_clock.cc:648] HybridClock initialized: now 1781965508017644 us; error 62 us; skew 500 ppm
I20260620 14:25:08.018848 10429 webserver.cc:460] Webserver started at http://127.0.98.3:44527/ using document root <none> and password file <none>
I20260620 14:25:08.019093 10429 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:08.019167 10429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:08.019317 10429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:08.020359 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "b92fdc0a131f411bbba3420d35f9e838"
format_stamp: "Formatted at 2026-06-20 14:25:08 on dist-test-slave-hfc3"
server_key: "33c86fed8a0c7cc6915adfbc1d3fe537"
server_key_iv: "2e9fa8f4a0ab7ecd54fa6814bd1b4424"
server_key_version: "encryptionkey@0"
I20260620 14:25:08.020844 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b92fdc0a131f411bbba3420d35f9e838"
format_stamp: "Formatted at 2026-06-20 14:25:08 on dist-test-slave-hfc3"
server_key: "33c86fed8a0c7cc6915adfbc1d3fe537"
server_key_iv: "2e9fa8f4a0ab7ecd54fa6814bd1b4424"
server_key_version: "encryptionkey@0"
I20260620 14:25:08.022141 10429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:25:08.022902 10443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:08.023061 10429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:08.023160 10429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "b92fdc0a131f411bbba3420d35f9e838"
format_stamp: "Formatted at 2026-06-20 14:25:08 on dist-test-slave-hfc3"
server_key: "33c86fed8a0c7cc6915adfbc1d3fe537"
server_key_iv: "2e9fa8f4a0ab7ecd54fa6814bd1b4424"
server_key_version: "encryptionkey@0"
I20260620 14:25:08.023236 10429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:08.026501 10429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:08.026765 10429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:08.026906 10429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:08.027107 10429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:08.027402 10429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:08.027456 10429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:08.027495 10429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:08.027523 10429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:08.033417 10429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39073
I20260620 14:25:08.033473 10556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39073 every 8 connection(s)
I20260620 14:25:08.033818 10429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:25:08.036746 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:08.034664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35931 (local address 127.0.98.62:38145)
0620 14:25:08.034735 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:08.034737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:08.034946 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:25:08.035373 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:08.035374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:08.035379 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:08.035439 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:08.036528 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:08.036530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:08.036530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:25:08.036534 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:08.036550 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:25:08.036554 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:08.036611 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:08.036659 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:25:08.036671 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:25:08.036969 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:08.034646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38145 (local address 127.0.98.3:35931)
0620 14:25:08.034823 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:08.034862 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:08.035295 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:08.035488 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:08.035492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:08.035499 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:08.036375 (+ 876us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:25:08.036376 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:08.036403 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:25:08.036432 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:08.036610 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:08.036613 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:08.036615 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:25:08.036665 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:25:08.036712 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260620 14:25:08.037467 10557 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38145
I20260620 14:25:08.037547 10557 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:08.037659 10557 heartbeater.cc:511] Master 127.0.98.62:38145 requested a full tablet report, sending...
I20260620 14:25:08.038000 10107 ts_manager.cc:194] Registered new tserver with Master: b92fdc0a131f411bbba3420d35f9e838 (127.0.98.3:39073)
I20260620 14:25:08.041461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 10429
I20260620 14:25:08.041545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:25:08.041699 392 external_mini_cluster.cc:1428] Setting key 19e245c7a02656ecbb70f5963715cf1d
I20260620 14:25:08.042660 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:25:08.044512 10147 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:08.044032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.0.98.62:38145)
0620 14:25:08.044095 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:08.044097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:08.044245 (+ 148us) server_negotiation.cc:407] Connection header received
0620 14:25:08.044302 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:08.044302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:08.044307 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:08.044345 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:08.044438 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56708: BlockingRecv error: recv got EOF from 127.0.0.1:56708 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:25:08.044606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 10167
I20260620 14:25:08.049957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 10298
I20260620 14:25:08.054008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 10429
I20260620 14:25:08.058090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 10076
2026-06-20T14:25:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (564 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:25:08 dist-test-slave-hfc3 krb5kdc[10567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:25:08 dist-test-slave-hfc3 krb5kdc[10567](info): set up 2 sockets
Jun 20 14:25:08 dist-test-slave-hfc3 krb5kdc[10567](info): commencing operation
krb5kdc: starting...
W20260620 14:25:09.087407 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:25:09 dist-test-slave-hfc3 krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:25:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:25:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:25:09.204583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:09.284575 10580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:09.284785 10580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:09.284807 10580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:09.285969 10580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:09.286013 10580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:09.286027 10580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:09.286041 10580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:09.286098 10580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:09.287626 10580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:25:09.287892 10580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:09.288125 10580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:09.292990 10586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:09.293221 10585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:09.293226 10580 server_base.cc:1034] running on GCE node
W20260620 14:25:09.293228 10588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:09.293532 10580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:09.293772 10580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:09.294926 10580 hybrid_clock.cc:648] HybridClock initialized: now 1781965509294911 us; error 31 us; skew 500 ppm
Jun 20 14:25:09 dist-test-slave-hfc3 krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:09.297324 10580 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:25:09.297873 10580 webserver.cc:460] Webserver started at http://127.0.98.62:36109/ using document root <none> and password file <none>
I20260620 14:25:09.298158 10580 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:09.298231 10580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:09.298389 10580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:09.299538 10580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "3d35fffdddeb4d319e21134e9d66cac2"
format_stamp: "Formatted at 2026-06-20 14:25:09 on dist-test-slave-hfc3"
server_key: "f628477f96a41c4be308aeeaefaf08a2"
server_key_iv: "9f7b924008afdd320c50bfa98202e03b"
server_key_version: "encryptionkey@0"
I20260620 14:25:09.299891 10580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d35fffdddeb4d319e21134e9d66cac2"
format_stamp: "Formatted at 2026-06-20 14:25:09 on dist-test-slave-hfc3"
server_key: "f628477f96a41c4be308aeeaefaf08a2"
server_key_iv: "9f7b924008afdd320c50bfa98202e03b"
server_key_version: "encryptionkey@0"
I20260620 14:25:09.301263 10580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:25:09.301990 10595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:09.302166 10580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:25:09.302266 10580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "3d35fffdddeb4d319e21134e9d66cac2"
format_stamp: "Formatted at 2026-06-20 14:25:09 on dist-test-slave-hfc3"
server_key: "f628477f96a41c4be308aeeaefaf08a2"
server_key_iv: "9f7b924008afdd320c50bfa98202e03b"
server_key_version: "encryptionkey@0"
I20260620 14:25:09.302367 10580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:09.315984 10580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:09.317862 10580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:09.318024 10580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:09.322067 10580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45301
I20260620 14:25:09.322101 10647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45301 every 8 connection(s)
I20260620 14:25:09.322506 10580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:25:09.323486 10648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:09.325664 10648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2: Bootstrap starting.
I20260620 14:25:09.326187 10648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:09.326452 10648 log.cc:824] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:09.327065 10648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2: No bootstrap required, opened a new log
I20260620 14:25:09.328563 10648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d35fffdddeb4d319e21134e9d66cac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45301 } }
I20260620 14:25:09.328698 10648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:09.328722 10648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d35fffdddeb4d319e21134e9d66cac2, State: Initialized, Role: FOLLOWER
I20260620 14:25:09.328856 10648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d35fffdddeb4d319e21134e9d66cac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45301 } }
I20260620 14:25:09.328938 10648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:25:09.328966 10648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:25:09.329001 10648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:09.329552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 10580
I20260620 14:25:09.329674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:25:09.329901 392 external_mini_cluster.cc:1428] Setting key dc026d55bc8e3661c92284c0c5852288
I20260620 14:25:09.329988 10648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d35fffdddeb4d319e21134e9d66cac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45301 } }
I20260620 14:25:09.330097 10648 leader_election.cc:302] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d35fffdddeb4d319e21134e9d66cac2; no voters:
I20260620 14:25:09.330260 10648 leader_election.cc:288] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:25:09.330283 10651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:09.330418 10651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [term 1 LEADER]: Becoming Leader. State: Replica: 3d35fffdddeb4d319e21134e9d66cac2, State: Running, Role: LEADER
I20260620 14:25:09.330503 10651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d35fffdddeb4d319e21134e9d66cac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45301 } }
I20260620 14:25:09.330577 10648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:09.330816 10651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d35fffdddeb4d319e21134e9d66cac2. Latest consensus state: current_term: 1 leader_uuid: "3d35fffdddeb4d319e21134e9d66cac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d35fffdddeb4d319e21134e9d66cac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45301 } } }
I20260620 14:25:09.330904 10651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:09.330832 10653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d35fffdddeb4d319e21134e9d66cac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d35fffdddeb4d319e21134e9d66cac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45301 } } }
I20260620 14:25:09.330956 10653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:09.331799 10657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:25:09.332305 10657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:25:09.333863 10657 catalog_manager.cc:1347] Generated new cluster ID: 885fe5d794bb4cb6b19eca8d859b06d1
I20260620 14:25:09.333916 10657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:25:09 dist-test-slave-hfc3 krb5kdc[10567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:09.340741 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.330555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.0.98.62:45301)
0620 14:25:09.330864 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:09.330874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:09.331128 (+ 254us) server_negotiation.cc:407] Connection header received
0620 14:25:09.331261 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:09.331269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:09.331324 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:09.331480 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:09.332309 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.333499 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.334458 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.334880 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.335016 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.337347 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:09.337351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:09.337352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:09.337363 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:09.339097 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:09.339463 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:09.339466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:09.339467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:09.339567 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:09.339797 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:09.339798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:09.339798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:09.340084 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:09.340248 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:09.340365 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:25:09.340514 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":73,"threads_started":1}
I20260620 14:25:09.355707 10657 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:25:09.356467 10657 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:25:09.362938 10657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2: Generated new TSK 0
I20260620 14:25:09.363147 10657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:25:09.392654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45301
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:09.473785 10674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:09.474052 10674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:09.474094 10674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:09.475275 10674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:09.475348 10674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:09.475394 10674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:25:09.476878 10674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:25:09.477169 10674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:09.477409 10674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:09.482372 10680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:09.482374 10682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:09.482467 10674 server_base.cc:1034] running on GCE node
W20260620 14:25:09.482400 10679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:09.482816 10674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:09.483083 10674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:09.484241 10674 hybrid_clock.cc:648] HybridClock initialized: now 1781965509484234 us; error 37 us; skew 500 ppm
Jun 20 14:25:09 dist-test-slave-hfc3 krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:09.486723 10674 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:25:09.487150 10674 webserver.cc:460] Webserver started at http://127.0.98.1:37469/ using document root <none> and password file <none>
I20260620 14:25:09.487433 10674 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:09.487498 10674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:09.487630 10674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:09.488629 10674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "6519e970557948689cc908617ce8b2e9"
format_stamp: "Formatted at 2026-06-20 14:25:09 on dist-test-slave-hfc3"
server_key: "eb4b24988bfcd49c405712f4d2c90f24"
server_key_iv: "deb0b0e8d17df84d4f0162d5da0a0ba9"
server_key_version: "encryptionkey@0"
I20260620 14:25:09.488977 10674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6519e970557948689cc908617ce8b2e9"
format_stamp: "Formatted at 2026-06-20 14:25:09 on dist-test-slave-hfc3"
server_key: "eb4b24988bfcd49c405712f4d2c90f24"
server_key_iv: "deb0b0e8d17df84d4f0162d5da0a0ba9"
server_key_version: "encryptionkey@0"
I20260620 14:25:09.490432 10674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:25:09.491164 10689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:09.491330 10674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:25:09.491427 10674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "6519e970557948689cc908617ce8b2e9"
format_stamp: "Formatted at 2026-06-20 14:25:09 on dist-test-slave-hfc3"
server_key: "eb4b24988bfcd49c405712f4d2c90f24"
server_key_iv: "deb0b0e8d17df84d4f0162d5da0a0ba9"
server_key_version: "encryptionkey@0"
I20260620 14:25:09.491511 10674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:09.506515 10674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:09.508888 10674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:09.509055 10674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:09.509272 10674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:09.509573 10674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:09.509624 10674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:09.509667 10674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:09.509692 10674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:09.515609 10674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44931
I20260620 14:25:09.515650 10802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44931 every 8 connection(s)
I20260620 14:25:09.516034 10674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:25:09.517961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 10674
I20260620 14:25:09.518061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:25:09.518285 392 external_mini_cluster.cc:1428] Setting key c1610eb2a1d6feb66a7d38def8e3250e
Jun 20 14:25:09 dist-test-slave-hfc3 krb5kdc[10567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:25:09.527053 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.516847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38909 (local address 127.0.98.62:45301)
0620 14:25:09.516925 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:09.516928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:09.517108 (+ 180us) server_negotiation.cc:407] Connection header received
0620 14:25:09.517772 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:09.517774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:09.517794 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:09.517960 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:09.519254 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.519733 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.521063 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.521388 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.521467 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.524105 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:09.524106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:09.524107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:09.524113 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:09.525536 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:09.526039 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:09.526040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:09.526040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:09.526094 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:09.526400 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:09.526401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:09.526401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:09.526510 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:09.526562 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:09.526910 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 14:25:09.526982 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:25:09.527328 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.516805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45301 (local address 127.0.98.1:38909)
0620 14:25:09.516964 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:09.517006 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:09.517652 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:09.518047 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:09.518052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:09.518870 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:09.519156 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:09.519162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.519834 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.519835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:09.520969 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:09.520972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.521477 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.521478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:09.521576 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.522138 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:09.522146 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:09.524010 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:09.525620 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:09.525628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:09.525630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:09.525955 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:09.526170 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:09.526171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:09.526172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:09.526305 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:09.526579 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:09.526585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:09.526760 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 14:25:09.526875 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:25:09.527047 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260620 14:25:09.528059 10803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45301
I20260620 14:25:09.528141 10803 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:09.528275 10803 heartbeater.cc:511] Master 127.0.98.62:45301 requested a full tablet report, sending...
I20260620 14:25:09.528775 10612 ts_manager.cc:194] Registered new tserver with Master: 6519e970557948689cc908617ce8b2e9 (127.0.98.1:44931)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:25:09.529573 10612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:25:09.543090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45301
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:09.628400 10808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:09.628618 10808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:09.628641 10808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:09.629789 10808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:09.629832 10808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:09.629859 10808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:25:09.631470 10808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:25:09.631726 10808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:09.631974 10808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:09.636934 10813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:09.636938 10816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:09.637001 10808 server_base.cc:1034] running on GCE node
W20260620 14:25:09.636984 10814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:09.637367 10808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:09.637655 10808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:09.638804 10808 hybrid_clock.cc:648] HybridClock initialized: now 1781965509638788 us; error 46 us; skew 500 ppm
Jun 20 14:25:09 dist-test-slave-hfc3 krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:09.641336 10808 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:25:09.641791 10808 webserver.cc:460] Webserver started at http://127.0.98.2:43339/ using document root <none> and password file <none>
I20260620 14:25:09.642020 10808 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:09.642088 10808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:09.642239 10808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:09.643332 10808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "4476c6bcccdb451b8c181467fbf69426"
format_stamp: "Formatted at 2026-06-20 14:25:09 on dist-test-slave-hfc3"
server_key: "a27ae0ec813b3677391e237a53af4756"
server_key_iv: "fb0633d6399a6ce9a95bbbcd7acaf8ea"
server_key_version: "encryptionkey@0"
I20260620 14:25:09.643714 10808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4476c6bcccdb451b8c181467fbf69426"
format_stamp: "Formatted at 2026-06-20 14:25:09 on dist-test-slave-hfc3"
server_key: "a27ae0ec813b3677391e237a53af4756"
server_key_iv: "fb0633d6399a6ce9a95bbbcd7acaf8ea"
server_key_version: "encryptionkey@0"
I20260620 14:25:09.645084 10808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:25:09.645794 10823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:09.645993 10808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:25:09.646090 10808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "4476c6bcccdb451b8c181467fbf69426"
format_stamp: "Formatted at 2026-06-20 14:25:09 on dist-test-slave-hfc3"
server_key: "a27ae0ec813b3677391e237a53af4756"
server_key_iv: "fb0633d6399a6ce9a95bbbcd7acaf8ea"
server_key_version: "encryptionkey@0"
I20260620 14:25:09.646173 10808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:09.671226 10808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:09.673769 10808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:09.673949 10808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:09.674220 10808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:09.674608 10808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:09.674665 10808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:09.674707 10808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:09.674737 10808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:09.680382 10808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33239
I20260620 14:25:09.680457 10936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33239 every 8 connection(s)
I20260620 14:25:09.680804 10808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:25:09 dist-test-slave-hfc3 krb5kdc[10567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:09.688652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 10808
I20260620 14:25:09.688764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:25:09.688975 392 external_mini_cluster.cc:1428] Setting key 8850cac6ab111c5d1334095079856d7c
I20260620 14:25:09.689677 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.681522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53041 (local address 127.0.98.62:45301)
0620 14:25:09.681568 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:09.681569 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:09.681767 (+ 198us) server_negotiation.cc:407] Connection header received
0620 14:25:09.682261 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:09.682262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:09.682271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:09.682375 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:09.683258 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.683748 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.684695 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.684873 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.684922 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.687052 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:09.687053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:09.687054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:09.687060 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:09.688248 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:09.688632 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:09.688633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:09.688633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:09.688700 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:09.689025 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:09.689026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:09.689026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:09.689154 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:09.689226 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:09.689503 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:25:09.689589 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:25:09.689961 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.681491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45301 (local address 127.0.98.2:53041)
0620 14:25:09.681637 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:09.681679 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:09.682159 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:09.682422 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:09.682425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:09.682983 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:09.683178 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:09.683183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.683828 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.683828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:09.684623 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:09.684624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.684942 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.684943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:09.685038 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.685580 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:09.685588 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:09.686971 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:09.688318 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:09.688323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:09.688325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:09.688539 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:09.688780 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:09.688781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:09.688782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:09.688927 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:09.689230 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:09.689237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:09.689372 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 14:25:09.689482 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:25:09.689791 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260620 14:25:09.690585 10937 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45301
I20260620 14:25:09.690690 10937 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:09.690881 10937 heartbeater.cc:511] Master 127.0.98.62:45301 requested a full tablet report, sending...
I20260620 14:25:09.691305 10612 ts_manager.cc:194] Registered new tserver with Master: 4476c6bcccdb451b8c181467fbf69426 (127.0.98.2:33239)
I20260620 14:25:09.691843 10612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53041
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:25:09.713593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45301
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:09.795751 10942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:09.795964 10942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:09.795986 10942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:09.797118 10942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:09.797160 10942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:09.797211 10942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:25:09.798789 10942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:25:09.799028 10942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:09.799248 10942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:09.804193 10948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:09.804191 10950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:09.804435 10942 server_base.cc:1034] running on GCE node
W20260620 14:25:09.804571 10947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:09.804798 10942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:09.805053 10942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:09.806213 10942 hybrid_clock.cc:648] HybridClock initialized: now 1781965509806202 us; error 40 us; skew 500 ppm
Jun 20 14:25:09 dist-test-slave-hfc3 krb5kdc[10567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:09.808787 10942 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:25:09.809203 10942 webserver.cc:460] Webserver started at http://127.0.98.3:40005/ using document root <none> and password file <none>
I20260620 14:25:09.809438 10942 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:09.809508 10942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:09.809659 10942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:09.810770 10942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "e6f484350c654000a2061aecb7dd9b61"
format_stamp: "Formatted at 2026-06-20 14:25:09 on dist-test-slave-hfc3"
server_key: "7986598d822ab19a7094fcc2224336e0"
server_key_iv: "13837a905403ce9f188694ab014885ab"
server_key_version: "encryptionkey@0"
I20260620 14:25:09.811133 10942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e6f484350c654000a2061aecb7dd9b61"
format_stamp: "Formatted at 2026-06-20 14:25:09 on dist-test-slave-hfc3"
server_key: "7986598d822ab19a7094fcc2224336e0"
server_key_iv: "13837a905403ce9f188694ab014885ab"
server_key_version: "encryptionkey@0"
I20260620 14:25:09.812469 10942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:25:09.813169 10957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:09.813328 10942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:25:09.813432 10942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "e6f484350c654000a2061aecb7dd9b61"
format_stamp: "Formatted at 2026-06-20 14:25:09 on dist-test-slave-hfc3"
server_key: "7986598d822ab19a7094fcc2224336e0"
server_key_iv: "13837a905403ce9f188694ab014885ab"
server_key_version: "encryptionkey@0"
I20260620 14:25:09.813509 10942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:09.834304 10942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:09.836802 10942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:09.836975 10942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:09.837218 10942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:09.837548 10942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:09.837604 10942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:09.837647 10942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:09.837694 10942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:09.844817 10942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44211
I20260620 14:25:09.844870 11070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44211 every 8 connection(s)
I20260620 14:25:09.845230 10942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:25:09.849068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 10942
I20260620 14:25:09.849144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:25:09.849326 392 external_mini_cluster.cc:1428] Setting key 53ac73a7a8009bb05abed6e808691cca
Jun 20 14:25:09 dist-test-slave-hfc3 krb5kdc[10567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:09.854787 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.845985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42531 (local address 127.0.98.62:45301)
0620 14:25:09.846051 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:09.846054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:09.846266 (+ 212us) server_negotiation.cc:407] Connection header received
0620 14:25:09.846895 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:09.846896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:09.846908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:09.846976 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:09.847949 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.848504 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.849797 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.850114 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.850199 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.852443 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:09.852444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:09.852444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:09.852451 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:09.853620 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:09.853995 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:09.853996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:09.853996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:09.854044 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:09.854252 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:09.854253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:09.854253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:09.854391 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:09.854450 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:09.854662 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 14:25:09.854723 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:25:09.855037 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.845971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45301 (local address 127.0.98.3:42531)
0620 14:25:09.846138 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:09.846175 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:09.846786 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:09.847053 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:09.847056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:09.847634 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:09.847849 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:09.847854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.848602 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.848602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:09.849665 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:09.849668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.850202 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.850204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:09.850391 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.850984 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:09.850992 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:09.852341 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:09.853691 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:09.853697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:09.853699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:09.853925 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:09.854100 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:09.854101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:09.854101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:09.854200 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:09.854441 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:09.854444 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:09.854554 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 14:25:09.854635 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:25:09.854752 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260620 14:25:09.855589 11071 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45301
I20260620 14:25:09.855697 11071 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:09.855876 11071 heartbeater.cc:511] Master 127.0.98.62:45301 requested a full tablet report, sending...
I20260620 14:25:09.856262 10612 ts_manager.cc:194] Registered new tserver with Master: e6f484350c654000a2061aecb7dd9b61 (127.0.98.3:44211)
I20260620 14:25:09.856657 10612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42531
I20260620 14:25:09.860993 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:25:09.868120 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.862269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38740 (local address 127.0.98.62:45301)
0620 14:25:09.862323 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:09.862325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:09.862459 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:25:09.862506 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:09.862507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:09.862515 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:09.862566 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:09.863116 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.863591 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.864303 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.864487 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.864543 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.865750 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:09.865751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:09.865751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:09.865756 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:09.867176 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:09.867483 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:09.867483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:09.867484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:09.867532 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:09.867695 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:09.867696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:09.867696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:09.867831 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:09.867902 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:09.868000 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:25:09.868058 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:25:09.869712 10612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:25:09.871299 10612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:25:09.877918 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.873776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39272 (local address 127.0.98.3:44211)
0620 14:25:09.873999 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:09.874005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:09.874018 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:25:09.874073 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:09.874075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:09.874091 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:09.874189 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:09.874434 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.875582 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.876914 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.877471 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.877563 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.877619 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:09.877766 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 14:25:09.877810 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
I20260620 14:25:09.877934 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.873605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33239 (local address 127.0.0.1:51024)
0620 14:25:09.873950 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:09.873959 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:09.874017 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:09.874814 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:09.874815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:09.874817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:09.874867 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:09.874868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.875990 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.875991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:09.876850 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:09.876853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.877552 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.877554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:09.877633 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.877648 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:25:09.877750 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:25:09.877816 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":96,"threads_started":1}
I20260620 14:25:09.877934 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.873558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44211 (local address 127.0.0.1:39272)
0620 14:25:09.873756 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:09.873801 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:09.873873 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:09.874202 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:09.874205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:09.874207 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:09.874278 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:09.874280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.875695 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.875696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:09.876830 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:09.876832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.877580 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.877581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:09.877648 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.877651 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:25:09.877752 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:25:09.877829 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":29,"thread_start_us":67,"threads_started":1}
I20260620 14:25:09.878222 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.873746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.0.98.2:33239)
0620 14:25:09.874612 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:09.874617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:09.874627 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:25:09.874676 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:09.874678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:09.874693 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:09.874798 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:09.874927 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.875874 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.876967 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.877461 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.877525 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.877610 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:09.878101 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 14:25:09.878150 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":96,"threads_started":1}
I20260620 14:25:09.879060 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.873764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44931 (local address 127.0.0.1:50884)
0620 14:25:09.874153 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:09.874164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:09.874209 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:09.874428 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:09.874429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:09.874431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:09.874483 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:09.874484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.875503 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.875504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:09.876902 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:09.876904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.878802 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.878804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:09.878866 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.878871 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:25:09.878941 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:25:09.878986 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":30,"thread_start_us":61,"threads_started":1}
I20260620 14:25:09.878997 10990 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd7013cb73f4249a8a9c0f0661e3fe1 (DEFAULT_TABLE table=test-table [id=8fd24fb5569b4c0988407192dc4cb812]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:09.879021 10871 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd7013cb73f4249a8a9c0f0661e3fe1 (DEFAULT_TABLE table=test-table [id=8fd24fb5569b4c0988407192dc4cb812]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:09.879082 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:09.873977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.0.98.1:44931)
0620 14:25:09.874180 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:09.874184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:09.874198 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:25:09.874276 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:09.874278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:09.874294 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:09.874433 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:09.874550 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.875388 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.877646 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:09.878716 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:09.878791 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:09.878844 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:09.878953 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:25:09.878998 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
I20260620 14:25:09.879352 10990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fd7013cb73f4249a8a9c0f0661e3fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:09.879369 10871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fd7013cb73f4249a8a9c0f0661e3fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:09.880155 10737 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd7013cb73f4249a8a9c0f0661e3fe1 (DEFAULT_TABLE table=test-table [id=8fd24fb5569b4c0988407192dc4cb812]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:09.880486 10737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fd7013cb73f4249a8a9c0f0661e3fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:09.881919 11088 tablet_bootstrap.cc:492] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61: Bootstrap starting.
I20260620 14:25:09.882665 11088 tablet_bootstrap.cc:654] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:09.882694 11089 tablet_bootstrap.cc:492] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426: Bootstrap starting.
I20260620 14:25:09.883272 11088 log.cc:824] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:09.883378 11089 tablet_bootstrap.cc:654] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:09.883608 11090 tablet_bootstrap.cc:492] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9: Bootstrap starting.
I20260620 14:25:09.883944 11089 log.cc:824] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:09.884207 11090 tablet_bootstrap.cc:654] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:09.884591 11089 tablet_bootstrap.cc:492] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426: No bootstrap required, opened a new log
I20260620 14:25:09.884673 11089 ts_tablet_manager.cc:1397] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:09.884706 11090 log.cc:824] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:09.885401 11090 tablet_bootstrap.cc:492] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9: No bootstrap required, opened a new log
I20260620 14:25:09.885411 11088 tablet_bootstrap.cc:492] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61: No bootstrap required, opened a new log
I20260620 14:25:09.885476 11088 ts_tablet_manager.cc:1397] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 14:25:09.885483 11090 ts_tablet_manager.cc:1397] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:25:09.886487 11089 raft_consensus.cc:348] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f484350c654000a2061aecb7dd9b61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44211 } } peers { permanent_uuid: "6519e970557948689cc908617ce8b2e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44931 } } peers { permanent_uuid: "4476c6bcccdb451b8c181467fbf69426" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } }
I20260620 14:25:09.886651 11089 raft_consensus.cc:374] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:09.886694 11089 raft_consensus.cc:729] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4476c6bcccdb451b8c181467fbf69426, State: Initialized, Role: FOLLOWER
I20260620 14:25:09.887430 11090 raft_consensus.cc:348] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f484350c654000a2061aecb7dd9b61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44211 } } peers { permanent_uuid: "6519e970557948689cc908617ce8b2e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44931 } } peers { permanent_uuid: "4476c6bcccdb451b8c181467fbf69426" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } }
I20260620 14:25:09.887430 11088 raft_consensus.cc:348] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f484350c654000a2061aecb7dd9b61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44211 } } peers { permanent_uuid: "6519e970557948689cc908617ce8b2e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44931 } } peers { permanent_uuid: "4476c6bcccdb451b8c181467fbf69426" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } }
I20260620 14:25:09.887601 11088 raft_consensus.cc:374] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:09.887601 11090 raft_consensus.cc:374] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:09.887645 11088 raft_consensus.cc:729] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f484350c654000a2061aecb7dd9b61, State: Initialized, Role: FOLLOWER
I20260620 14:25:09.887674 11090 raft_consensus.cc:729] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6519e970557948689cc908617ce8b2e9, State: Initialized, Role: FOLLOWER
I20260620 14:25:09.887779 11090 consensus_queue.cc:260] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f484350c654000a2061aecb7dd9b61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44211 } } peers { permanent_uuid: "6519e970557948689cc908617ce8b2e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44931 } } peers { permanent_uuid: "4476c6bcccdb451b8c181467fbf69426" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } }
I20260620 14:25:09.887777 11088 consensus_queue.cc:260] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f484350c654000a2061aecb7dd9b61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44211 } } peers { permanent_uuid: "6519e970557948689cc908617ce8b2e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44931 } } peers { permanent_uuid: "4476c6bcccdb451b8c181467fbf69426" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } }
I20260620 14:25:09.887984 11089 consensus_queue.cc:260] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f484350c654000a2061aecb7dd9b61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44211 } } peers { permanent_uuid: "6519e970557948689cc908617ce8b2e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44931 } } peers { permanent_uuid: "4476c6bcccdb451b8c181467fbf69426" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } }
I20260620 14:25:09.888054 11088 ts_tablet_manager.cc:1428] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 14:25:09.888054 11090 ts_tablet_manager.cc:1428] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:25:09.888175 10803 heartbeater.cc:503] Master 127.0.98.62:45301 was elected leader, sending a full tablet report...
I20260620 14:25:09.888315 11089 ts_tablet_manager.cc:1428] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260620 14:25:09.888329 10937 heartbeater.cc:503] Master 127.0.98.62:45301 was elected leader, sending a full tablet report...
I20260620 14:25:09.888506 11071 heartbeater.cc:503] Master 127.0.98.62:45301 was elected leader, sending a full tablet report...
W20260620 14:25:09.931418 10938 tablet.cc:2369] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:25:10.016883 10804 tablet.cc:2369] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:25:10.021195 11095 raft_consensus.cc:482] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:25:10.021302 11095 raft_consensus.cc:504] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f484350c654000a2061aecb7dd9b61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44211 } } peers { permanent_uuid: "6519e970557948689cc908617ce8b2e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44931 } } peers { permanent_uuid: "4476c6bcccdb451b8c181467fbf69426" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } }
I20260620 14:25:10.021742 11095 leader_election.cc:288] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6519e970557948689cc908617ce8b2e9 (127.0.98.1:44931), 4476c6bcccdb451b8c181467fbf69426 (127.0.98.2:33239)
I20260620 14:25:10.025285 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:10.022075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44905 (local address 127.0.98.2:33239)
0620 14:25:10.022122 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:10.022125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:10.022217 (+ 92us) server_negotiation.cc:407] Connection header received
0620 14:25:10.022245 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:10.022246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:10.022258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:10.022315 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:10.022507 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:10.023246 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:10.024382 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:10.024948 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:10.025011 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:10.025020 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:10.025175 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:25:10.025214 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:25:10.025285 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:10.022078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57043 (local address 127.0.98.1:44931)
0620 14:25:10.022122 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:10.022125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:10.022136 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:25:10.022168 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:10.022169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:10.022181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:10.022266 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:10.022497 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:10.023247 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:10.024343 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:10.024905 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:10.024991 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:10.025004 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:10.025176 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:25:10.025217 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:25:10.025297 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:10.021964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44931 (local address 127.0.98.3:57043)
0620 14:25:10.022027 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:10.022040 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:10.022085 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:10.022282 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:10.022283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:10.022285 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:10.022394 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:10.022396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:10.023315 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:10.023316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:10.024242 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:10.024244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:10.025002 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:10.025003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:10.025090 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:10.025093 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:25:10.025153 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:25:10.025200 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":63}
I20260620 14:25:10.025303 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:10.021986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33239 (local address 127.0.98.3:44905)
0620 14:25:10.022144 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:10.022153 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:10.022206 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:10.022326 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:10.022327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:10.022350 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:10.022413 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:10.022414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:10.023364 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:10.023365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:10.024238 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:10.024242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:10.025023 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:10.025024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:10.025090 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:10.025094 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:25:10.025153 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:25:10.025200 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":7,"thread_start_us":100,"threads_started":1}
I20260620 14:25:10.025720 10757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd7013cb73f4249a8a9c0f0661e3fe1" candidate_uuid: "e6f484350c654000a2061aecb7dd9b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6519e970557948689cc908617ce8b2e9" is_pre_election: true
I20260620 14:25:10.025923 10757 raft_consensus.cc:2454] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f484350c654000a2061aecb7dd9b61 in term 0.
I20260620 14:25:10.025923 10891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd7013cb73f4249a8a9c0f0661e3fe1" candidate_uuid: "e6f484350c654000a2061aecb7dd9b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4476c6bcccdb451b8c181467fbf69426" is_pre_election: true
I20260620 14:25:10.026043 10891 raft_consensus.cc:2454] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f484350c654000a2061aecb7dd9b61 in term 0.
I20260620 14:25:10.026145 10958 leader_election.cc:302] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6519e970557948689cc908617ce8b2e9, e6f484350c654000a2061aecb7dd9b61; no voters:
I20260620 14:25:10.026311 11095 raft_consensus.cc:2781] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:25:10.026369 11095 raft_consensus.cc:482] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:25:10.026386 11095 raft_consensus.cc:3037] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:10.027058 11095 raft_consensus.cc:504] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f484350c654000a2061aecb7dd9b61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44211 } } peers { permanent_uuid: "6519e970557948689cc908617ce8b2e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44931 } } peers { permanent_uuid: "4476c6bcccdb451b8c181467fbf69426" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } }
I20260620 14:25:10.027204 11095 leader_election.cc:288] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [CANDIDATE]: Term 1 election: Requested vote from peers 6519e970557948689cc908617ce8b2e9 (127.0.98.1:44931), 4476c6bcccdb451b8c181467fbf69426 (127.0.98.2:33239)
I20260620 14:25:10.027361 10891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd7013cb73f4249a8a9c0f0661e3fe1" candidate_uuid: "e6f484350c654000a2061aecb7dd9b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4476c6bcccdb451b8c181467fbf69426"
I20260620 14:25:10.027357 10757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd7013cb73f4249a8a9c0f0661e3fe1" candidate_uuid: "e6f484350c654000a2061aecb7dd9b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6519e970557948689cc908617ce8b2e9"
I20260620 14:25:10.027434 10757 raft_consensus.cc:3037] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:10.027434 10891 raft_consensus.cc:3037] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:10.028330 10757 raft_consensus.cc:2454] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f484350c654000a2061aecb7dd9b61 in term 1.
I20260620 14:25:10.028330 10891 raft_consensus.cc:2454] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f484350c654000a2061aecb7dd9b61 in term 1.
I20260620 14:25:10.028519 10959 leader_election.cc:302] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4476c6bcccdb451b8c181467fbf69426, e6f484350c654000a2061aecb7dd9b61; no voters:
I20260620 14:25:10.028628 11095 raft_consensus.cc:2781] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:10.028802 11095 raft_consensus.cc:686] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 1 LEADER]: Becoming Leader. State: Replica: e6f484350c654000a2061aecb7dd9b61, State: Running, Role: LEADER
I20260620 14:25:10.029011 11095 consensus_queue.cc:237] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f484350c654000a2061aecb7dd9b61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44211 } } peers { permanent_uuid: "6519e970557948689cc908617ce8b2e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44931 } } peers { permanent_uuid: "4476c6bcccdb451b8c181467fbf69426" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } }
I20260620 14:25:10.029863 10612 catalog_manager.cc:5495] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6f484350c654000a2061aecb7dd9b61 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e6f484350c654000a2061aecb7dd9b61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f484350c654000a2061aecb7dd9b61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6519e970557948689cc908617ce8b2e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4476c6bcccdb451b8c181467fbf69426" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } health_report { overall_health: UNKNOWN } } }
I20260620 14:25:10.064158 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:10.061543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39282 (local address 127.0.98.3:44211)
0620 14:25:10.061618 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:10.061620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:10.061628 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:25:10.061676 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:10.061676 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:10.061697 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:10.061778 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:10.061945 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:10.062541 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:10.063311 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:10.063527 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:10.063601 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:10.063786 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:10.063903 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:10.064011 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:10.064030 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 14:25:10.064051 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:25:10.066038 10891 raft_consensus.cc:1261] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426 [term 1 FOLLOWER]: Refusing update from remote peer e6f484350c654000a2061aecb7dd9b61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:10.066043 10757 raft_consensus.cc:1261] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 6519e970557948689cc908617ce8b2e9 [term 1 FOLLOWER]: Refusing update from remote peer e6f484350c654000a2061aecb7dd9b61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:10.066428 11098 consensus_queue.cc:1035] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4476c6bcccdb451b8c181467fbf69426" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:10.066587 11095 consensus_queue.cc:1035] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6519e970557948689cc908617ce8b2e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:10.068866 11101 mvcc.cc:205] Tried to move back new op lower bound from 7298930729227489280 to 7298930729079447552. Current Snapshot: MvccSnapshot[applied={T|T < 7298930729227489280}]
I20260620 14:25:10.069737 11103 mvcc.cc:205] Tried to move back new op lower bound from 7298930729227489280 to 7298930729079447552. Current Snapshot: MvccSnapshot[applied={T|T < 7298930729227489280 or (T in {7298930729227489280})}]
I20260620 14:25:10.071867 10612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:10.071969 10612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:10.072724 10612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d35fffdddeb4d319e21134e9d66cac2: Sending DeleteTablet for 3 replicas of tablet 0fd7013cb73f4249a8a9c0f0661e3fe1
I20260620 14:25:10.073065 10990 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd7013cb73f4249a8a9c0f0661e3fe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39272
I20260620 14:25:10.073086 10737 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd7013cb73f4249a8a9c0f0661e3fe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50884
I20260620 14:25:10.073104 10871 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd7013cb73f4249a8a9c0f0661e3fe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51024
I20260620 14:25:10.073304 11111 tablet_replica.cc:330] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61: stopping tablet replica
I20260620 14:25:10.073400 11111 raft_consensus.cc:2229] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:25:10.073526 11111 raft_consensus.cc:2258] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:10.073578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 10674
I20260620 14:25:10.073968 11111 ts_tablet_manager.cc:1905] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:10.076120 11111 ts_tablet_manager.cc:1918] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:25:10.076200 11111 log.cc:1196] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/0fd7013cb73f4249a8a9c0f0661e3fe1
I20260620 14:25:10.076474 11111 ts_tablet_manager.cc:1939] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P e6f484350c654000a2061aecb7dd9b61: Deleting consensus metadata
I20260620 14:25:10.076889 11113 tablet_replica.cc:330] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426: stopping tablet replica
I20260620 14:25:10.076905 10596 catalog_manager.cc:4841] TS e6f484350c654000a2061aecb7dd9b61 (127.0.98.3:44211): tablet 0fd7013cb73f4249a8a9c0f0661e3fe1 (table test-table [id=8fd24fb5569b4c0988407192dc4cb812]) successfully deleted
I20260620 14:25:10.077008 11113 raft_consensus.cc:2229] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:10.077116 11113 raft_consensus.cc:2258] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:10.077592 11113 ts_tablet_manager.cc:1905] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:10.079217 11113 ts_tablet_manager.cc:1918] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:25:10.079296 11113 log.cc:1196] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/0fd7013cb73f4249a8a9c0f0661e3fe1
I20260620 14:25:10.079577 11113 ts_tablet_manager.cc:1939] T 0fd7013cb73f4249a8a9c0f0661e3fe1 P 4476c6bcccdb451b8c181467fbf69426: Deleting consensus metadata
I20260620 14:25:10.079983 10597 catalog_manager.cc:4841] TS 4476c6bcccdb451b8c181467fbf69426 (127.0.98.2:33239): tablet 0fd7013cb73f4249a8a9c0f0661e3fe1 (table test-table [id=8fd24fb5569b4c0988407192dc4cb812]) successfully deleted
W20260620 14:25:10.081043 10596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44931 (error 108)
I20260620 14:25:10.081239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 10808
I20260620 14:25:10.082217 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:10.082092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44931 (local address 127.0.0.1:50888)
0620 14:25:10.082140 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:10.082176 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:25:10.082355 10596 catalog_manager.cc:4584] TS 6519e970557948689cc908617ce8b2e9 (127.0.98.1:44931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fd7013cb73f4249a8a9c0f0661e3fe1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44931: connect: Connection refused (error 111)
I20260620 14:25:10.086010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 10942
I20260620 14:25:10.090788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 10580
2026-06-20T14:25:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2032 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:25:10 dist-test-slave-hfc3 krb5kdc[11117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:25:10 dist-test-slave-hfc3 krb5kdc[11117](info): set up 2 sockets
Jun 20 14:25:10 dist-test-slave-hfc3 krb5kdc[11117](info): commencing operation
krb5kdc: starting...
W20260620 14:25:11.127967 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:25:11 dist-test-slave-hfc3 krb5kdc[11117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:25:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:25:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 14:25:11.288432 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLb3bhr/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/postgres -l logfile start
2026-06-20 14:25:13.716 UTC [11147] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:25:13.716 UTC [11147] LOG: listening on IPv4 address "127.0.98.20", port 59615
2026-06-20 14:25:13.720 UTC [11147] LOG: listening on Unix socket "/tmp/.s.PGSQL.59615"
2026-06-20 14:25:13.741 UTC [11150] LOG: database system was shut down at 2026-06-20 14:25:12 UTC
2026-06-20 14:25:13.744 UTC [11147] LOG: database system is ready to accept connections
I20260620 14:25:14.695422 392 mini_postgres.cc:96] Postgres bound to 59615
2026-06-20 14:25:14.701 UTC [11158] FATAL: database "slave" does not exist
127.0.98.20:59615 - accepting connections
I20260620 14:25:14.702179 392 mini_ranger.cc:161] Starting Ranger...
I20260620 14:25:14.716599 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:25:15.175807 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:25:15.175899 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-admin
2026-06-20 14:25:15,475 [I] DB FLAVOR :POSTGRES
2026-06-20 14:25:15,476 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:25:15,476 [I] Checking connection..
2026-06-20 14:25:15,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:25:15,724 [I] Checking connection passed.
2026-06-20 14:25:15,724 [I] --------- Verifying version history table ---------
2026-06-20 14:25:15,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:25:15,957 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:25:15,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:25:16,186 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:25:16,187 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:25:16,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:25:16.418 UTC [11222] WARNING: there is no transaction in progress
2026-06-20 14:25:16,424 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:25:16,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:25:16,655 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:25:16,656 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:25:16,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:25:16,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:25:17,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hfc3.c.gcp-upstream.internal','N') ;"
2026-06-20 14:25:17,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:25:17,613 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:25:17,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:25:17,850 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:25:17,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:25:18,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:25:18,332 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:25:18,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:25:19.282 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.297 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.313 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.327 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.342 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.360 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.414 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.421 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.435 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.445 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.456 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.470 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.479 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.485 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.490 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:19.496 UTC [11330] WARNING: there is no transaction in progress
2026-06-20 14:25:20,110 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:25:20,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:25:20,349 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:25:20,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:25:20,612 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:25:20,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:25:20,842 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:25:20,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:25:21,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:25:21,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hfc3.c.gcp-upstream.internal';"
2026-06-20 14:25:21,572 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:25:21,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:25:21,834 [I] DB_PATCHES have already been applied
I20260620 14:25:21.840400 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLb3bhr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:25:21.840466 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 14:25:21.843726 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:33151
Jun 20, 2026 2:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:25:22.946652 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.244s user 0.006s sys 0.004s
Jun 20, 2026 2:25:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:25:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Jun 20, 2026 2:25:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:25:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:25:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:25:45.132117 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:25:45 dist-test-slave-hfc3 krb5kdc[11117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965545, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:25:45.216442 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:25:45.226060 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:25:46.120226 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:25:46.120323 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-kms
I20260620 14:25:46.120348 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:59615
2026-06-20 14:25:46,616 [I] DB FLAVOR :POSTGRES
2026-06-20 14:25:46,617 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:25:46,617 [I] Checking connection
2026-06-20 14:25:46,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:25:46,914 [I] connection success
2026-06-20 14:25:46,914 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:25:46,914 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:25:46,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:25:47,172 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:25:47,172 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:25:47,172 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:25:47,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59615/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:25:47,445 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:25:47.921605 392 mini_ranger_kms.cc:310] Created kms service
I20260620 14:25:48.127291 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:25:48.174602 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:25:48 dist-test-slave-hfc3 krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965545, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:25:48.587504 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:25:48.587628 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:25:48.587649 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 14:25:48.591764 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11510
I20260620 14:25:48.591893 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54883
Jun 20, 2026 2:25:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:25:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:25:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:25:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:25:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:25:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 14:25:51 dist-test-slave-hfc3 krb5kdc[11117](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965551, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:25:51 dist-test-slave-hfc3 krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965551, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:25:52 dist-test-slave-hfc3 krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965551, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:26:22 dist-test-slave-hfc3 krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965551, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:26:22 dist-test-slave-hfc3 krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965551, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:26:52 dist-test-slave-hfc3 krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965551, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:26:52 dist-test-slave-hfc3 krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965551, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:27:22 dist-test-slave-hfc3 krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965551, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:27:22 dist-test-slave-hfc3 krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965551, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:27:48.596477 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 123.380s user 0.035s sys 0.585s
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1013: Failure
Failed
Bad status: Timed out: Failed to start the Ranger KMS service: process with PID 11510 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p11510
f196
n127.0.0.1:57990->127.0.98.20:59615'
I20260620 14:27:48.597425 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:27:48.628132 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:27:48.628244 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:27:48.972666 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:27:48.972 UTC [11147] LOG: received smart shutdown request
2026-06-20 14:27:48.975 UTC [11147] LOG: background worker "logical replication launcher" (PID 11156) exited with exit code 1
2026-06-20 14:27:48.976 UTC [11151] LOG: shutting down
2026-06-20 14:27:49.093 UTC [11147] LOG: database system is shut down
2026-06-20T14:27:49Z chronyd exiting
I20260620 14:27:49.105747 392 test_util.cc:175] -----------------------------------------------
I20260620 14:27:49.105930 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965454096380-392-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (158995 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:27:49 dist-test-slave-hfc3 krb5kdc[11644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:27:49 dist-test-slave-hfc3 krb5kdc[11644](info): set up 2 sockets
Jun 20 14:27:49 dist-test-slave-hfc3 krb5kdc[11644](info): commencing operation
krb5kdc: starting...
W20260620 14:27:50.125782 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:27:50 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:27:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:27:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 14:27:50.289214 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLb3bhr/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/postgres -l logfile start
2026-06-20 14:27:52.750 UTC [11674] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:27:52.750 UTC [11674] LOG: listening on IPv4 address "127.0.98.20", port 59687
2026-06-20 14:27:52.754 UTC [11674] LOG: listening on Unix socket "/tmp/.s.PGSQL.59687"
2026-06-20 14:27:52.790 UTC [11677] LOG: database system was shut down at 2026-06-20 14:27:51 UTC
2026-06-20 14:27:52.794 UTC [11674] LOG: database system is ready to accept connections
I20260620 14:27:53.833745 392 mini_postgres.cc:96] Postgres bound to 59687
2026-06-20 14:27:53.840 UTC [11685] FATAL: database "slave" does not exist
127.0.98.20:59687 - accepting connections
I20260620 14:27:53.840627 392 mini_ranger.cc:161] Starting Ranger...
I20260620 14:27:53.855705 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:27:54.296573 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:27:54.296660 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-admin
2026-06-20 14:27:54,579 [I] DB FLAVOR :POSTGRES
2026-06-20 14:27:54,580 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:27:54,580 [I] Checking connection..
2026-06-20 14:27:54,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:27:54,810 [I] Checking connection passed.
2026-06-20 14:27:54,810 [I] --------- Verifying version history table ---------
2026-06-20 14:27:54,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:27:55,068 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:27:55,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:27:55,411 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:27:55,411 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:27:55,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:27:55.664 UTC [11749] WARNING: there is no transaction in progress
2026-06-20 14:27:55,671 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:27:55,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:27:55,926 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:27:55,926 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:27:55,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:27:56,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:27:56,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hfc3.c.gcp-upstream.internal','N') ;"
2026-06-20 14:27:56,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:27:56,873 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:27:56,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:27:57,138 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:27:57,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:27:57,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:27:57,633 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:27:57,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:27:58.605 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.622 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.638 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.653 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.668 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.687 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.750 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.757 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.771 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.781 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.791 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.805 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.814 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.820 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.828 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:58.835 UTC [11857] WARNING: there is no transaction in progress
2026-06-20 14:27:59,470 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:27:59,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:27:59,730 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:27:59,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:27:59,967 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:27:59,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:28:00,233 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:28:00,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:28:00,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:28:00,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hfc3.c.gcp-upstream.internal';"
2026-06-20 14:28:00,951 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:28:00,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:28:01,206 [I] DB_PATCHES have already been applied
I20260620 14:28:01.212054 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLb3bhr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:28:01.212121 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 14:28:01.215543 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36315
Jun 20, 2026 2:28:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:28:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:28:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:28:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:28:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:28:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:28:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:28:02.371975 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.531s user 0.000s sys 0.009s
Jun 20, 2026 2:28:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:28:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 2:28:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:28:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:28:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:28:23.620239 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:28:23 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965703, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:28:23.717579 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:28:23.727447 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:28:24.595804 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:28:24.595897 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-kms
I20260620 14:28:24.595919 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:59687
2026-06-20 14:28:24,810 [I] DB FLAVOR :POSTGRES
2026-06-20 14:28:24,811 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:28:24,811 [I] Checking connection
2026-06-20 14:28:24,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:28:25,133 [I] connection success
2026-06-20 14:28:25,134 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:28:25,134 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:28:25,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:28:25,412 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:28:25,412 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:28:25,412 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:28:25,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59687/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:28:25,721 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:28:26.191180 392 mini_ranger_kms.cc:310] Created kms service
I20260620 14:28:26.398694 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:28:26.444913 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:28:26 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965703, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:26.861524 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:28:26.861656 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLb3bhr/build/release/bin/postgresql.jar:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLb3bhr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:28:26.861676 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 14:28:26.865702 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12037
I20260620 14:28:26.865787 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:34865
Jun 20, 2026 2:28:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:28:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:28:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:28:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:28:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:28:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:28:29.254320 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.537s user 0.003s sys 0.194s
I20260620 14:28:29.254482 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:28:29.254594 392 mini_ranger_kms.cc:401] 127.0.98.20:34865/kms/v1/keys
Jun 20 14:28:29 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965709, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:28:29 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965709, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:28:30 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965709, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:28:32 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:28:33.018544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34153
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34153
--ranger_config_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34865/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:33.123639 12088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:33.123914 12088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:33.123953 12088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:33.125102 12088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:33.125172 12088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:33.125197 12088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:33.125226 12088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:33.125257 12088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:33.127025 12088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:34865/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34153
--ranger_config_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:33.127305 12088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:33.127532 12088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:33.133508 12096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:33.133561 12094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:33.133523 12093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:33.133973 12088 server_base.cc:1034] running on GCE node
I20260620 14:28:33.134258 12088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:33.134620 12088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:33.135789 12088 hybrid_clock.cc:648] HybridClock initialized: now 1781965713135761 us; error 43 us; skew 500 ppm
Jun 20 14:28:33 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:33.138819 12088 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:33.139319 12088 webserver.cc:460] Webserver started at http://127.0.98.62:33823/ using document root <none> and password file <none>
I20260620 14:28:33.139549 12088 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:33.139613 12088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:33.139771 12088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:28:33 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:33.202268 12088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "693875695038486584e32e1731507c19"
format_stamp: "Formatted at 2026-06-20 14:28:33 on dist-test-slave-hfc3"
server_key: "7689103a550d57af9ae14838e7f50d68"
server_key_iv: "6d0f7280e5f478e1621fbb7ed2c8519d"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:33.202801 12088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "693875695038486584e32e1731507c19"
format_stamp: "Formatted at 2026-06-20 14:28:33 on dist-test-slave-hfc3"
server_key: "7689103a550d57af9ae14838e7f50d68"
server_key_iv: "6d0f7280e5f478e1621fbb7ed2c8519d"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:33.204411 12088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:28:33.219596 12103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:33.219828 12088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:33.219956 12088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "693875695038486584e32e1731507c19"
format_stamp: "Formatted at 2026-06-20 14:28:33 on dist-test-slave-hfc3"
server_key: "7689103a550d57af9ae14838e7f50d68"
server_key_iv: "6d0f7280e5f478e1621fbb7ed2c8519d"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:33.220053 12088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:33.234795 12088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:33.237198 12088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:33.237429 12088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:33.247578 12088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34153
I20260620 14:28:33.247671 12155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34153 every 8 connection(s)
I20260620 14:28:33.248099 12088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:33.248998 12156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:33.251859 12156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19: Bootstrap starting.
I20260620 14:28:33.252754 12156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:33.253299 12156 log.cc:824] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:33.254086 12156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19: No bootstrap required, opened a new log
I20260620 14:28:33.255795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 12088
I20260620 14:28:33.255942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:28:33.256162 392 external_mini_cluster.cc:1428] Setting key 5ca33a107f277d85b0cb6212cddf2742
I20260620 14:28:33.257682 12156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693875695038486584e32e1731507c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34153 } }
I20260620 14:28:33.257848 12156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:33.257912 12156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693875695038486584e32e1731507c19, State: Initialized, Role: FOLLOWER
I20260620 14:28:33.258076 12156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693875695038486584e32e1731507c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34153 } }
I20260620 14:28:33.258173 12156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:33.258296 12156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:33.258390 12156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:33.259071 12156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693875695038486584e32e1731507c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34153 } }
I20260620 14:28:33.259195 12156 leader_election.cc:302] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 693875695038486584e32e1731507c19; no voters:
I20260620 14:28:33.259501 12156 leader_election.cc:288] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:28:33.259575 12161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:33.259826 12161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 1 LEADER]: Becoming Leader. State: Replica: 693875695038486584e32e1731507c19, State: Running, Role: LEADER
I20260620 14:28:33.259943 12156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:33.259975 12161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693875695038486584e32e1731507c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34153 } }
I20260620 14:28:33.260413 12161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "693875695038486584e32e1731507c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693875695038486584e32e1731507c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34153 } } }
I20260620 14:28:33.260578 12161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:33.260860 12161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 693875695038486584e32e1731507c19. Latest consensus state: current_term: 1 leader_uuid: "693875695038486584e32e1731507c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693875695038486584e32e1731507c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34153 } } }
I20260620 14:28:33.261029 12161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:33.261257 12168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:33.262082 12168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:33.262396 12156 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 20 14:28:33 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:33.275065 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:33.256811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.0.98.62:34153)
0620 14:28:33.257237 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:33.257247 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:33.257292 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:28:33.257714 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:33.257724 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:33.257795 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:33.258841 (+ 1046us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:33.259015 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:33.260661 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:33.261612 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:33.262168 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:33.264228 (+ 2060us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:33.267924 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:33.267954 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:33.267954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:33.268093 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:33.272574 (+ 4481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:33.273075 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:33.273078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:33.273078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:33.273236 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:33.273660 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:33.273660 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:33.273661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:33.274001 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:33.274226 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:33.274296 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:28:33.274495 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":282,"threads_started":1}
I20260620 14:28:33.278820 12168 catalog_manager.cc:1347] Generated new cluster ID: 211ebb930d8f4485b25b3d4e66b49c7e
I20260620 14:28:33.278914 12168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:33.309823 12168 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:28:33.310792 12168 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:33.318313 12168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19: Generated new TSK 0
I20260620 14:28:33.318642 12168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:28:34.188932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34865/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34153
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:34.298873 12201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:34.299098 12201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:34.299124 12201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:34.300300 12201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:34.300343 12201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:34.300370 12201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:28:34.301920 12201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:34865/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:34.302183 12201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:34.302465 12201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:28:34.310448 12201 server_base.cc:1034] running on GCE node
W20260620 14:28:34.310418 12207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:34.310431 12206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:34.310592 12209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:34.310900 12201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:34.311291 12201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:34.312496 12201 hybrid_clock.cc:648] HybridClock initialized: now 1781965714312475 us; error 37 us; skew 500 ppm
Jun 20 14:28:34 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:34.316084 12201 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:28:34.316654 12201 webserver.cc:460] Webserver started at http://127.0.98.1:34999/ using document root <none> and password file <none>
I20260620 14:28:34.316905 12201 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:34.316977 12201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:34.317135 12201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:28:34 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:34.337149 12201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "592645c0b94b4337a1b16dbac8b77484"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-hfc3"
server_key: "4b2d8703e658e2a80a00946d0675a094"
server_key_iv: "cc1cda6cb6329c7ce4658c7f9002c163"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:34.337769 12201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "592645c0b94b4337a1b16dbac8b77484"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-hfc3"
server_key: "4b2d8703e658e2a80a00946d0675a094"
server_key_iv: "cc1cda6cb6329c7ce4658c7f9002c163"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:34.342365 12201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260620 14:28:34.356930 12216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.357178 12201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:28:34.357281 12201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "592645c0b94b4337a1b16dbac8b77484"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-hfc3"
server_key: "4b2d8703e658e2a80a00946d0675a094"
server_key_iv: "cc1cda6cb6329c7ce4658c7f9002c163"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:34.357384 12201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:34.381745 12201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:34.384680 12201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:34.384948 12201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:34.385322 12201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:34.385795 12201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:34.385888 12201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.385957 12201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:34.386008 12201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.403563 12201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42419
I20260620 14:28:34.404145 12201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:28:34.410084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 12201
I20260620 14:28:34.410207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:28:34.410440 392 external_mini_cluster.cc:1428] Setting key 6107ad29cc72c882202abe472c5f8abe
I20260620 14:28:34.412725 12331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42419 every 8 connection(s)
Jun 20 14:28:34 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:34.426062 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.413995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60781 (local address 127.0.98.62:34153)
0620 14:28:34.415078 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:34.415080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:34.415091 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:28:34.415411 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:34.415412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:34.415427 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:34.415520 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:34.416507 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.417269 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.418886 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.419201 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.419272 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.422282 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:34.422284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:34.422284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:34.422291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:34.424372 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:34.424886 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:34.424887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:34.424888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:34.424953 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:34.425247 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:34.425248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:34.425248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:34.425390 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:34.425431 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:34.425869 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 14:28:34.425959 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":80,"threads_started":1}
I20260620 14:28:34.426657 12335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.413589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34153 (local address 127.0.98.1:60781)
0620 14:28:34.414212 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:34.414273 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:34.415271 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:34.415664 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:34.415669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:34.416019 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:34.416388 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.416396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.417380 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.417381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.418786 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.418789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.419394 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.419395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.419571 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.420307 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:34.420319 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:34.422157 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:34.424479 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:34.424487 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:34.424490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:34.424794 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:34.425022 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:34.425022 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:34.425023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:34.425170 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:34.425478 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:34.425484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:34.425705 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 14:28:34.426173 (+ 468us) client_negotiation.cc:240] Negotiation successful
0620 14:28:34.426374 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":162,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:28:34.429379 12332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34153
I20260620 14:28:34.429582 12332 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:34.429844 12332 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
I20260620 14:28:34.430526 12120 ts_manager.cc:194] Registered new tserver with Master: 592645c0b94b4337a1b16dbac8b77484 (127.0.98.1:42419)
I20260620 14:28:34.431396 12120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:28:34.457744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34865/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34153
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:34.611677 12338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:34.612062 12338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:34.612165 12338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:34.613986 12338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:34.614091 12338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:34.614197 12338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:28:34.616652 12338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:34865/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:34.617182 12338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:34.617537 12338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:34.626092 12346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:34.627127 12338 server_base.cc:1034] running on GCE node
W20260620 14:28:34.627475 12348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:34.628363 12345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:34.628785 12338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:34.629120 12338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:34.630321 12338 hybrid_clock.cc:648] HybridClock initialized: now 1781965714630298 us; error 37 us; skew 500 ppm
Jun 20 14:28:34 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:34.634871 12338 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:28:34.635623 12338 webserver.cc:460] Webserver started at http://127.0.98.2:40881/ using document root <none> and password file <none>
I20260620 14:28:34.635972 12338 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:34.636118 12338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:34.636348 12338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:28:34 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:34.651755 12338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "dd9e4f405e1c46c68cebd1a1204c5053"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-hfc3"
server_key: "b24a876e62a60bd583947083044fff83"
server_key_iv: "7915e502f10d7dabeeb9942598e428b1"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:34.652325 12338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dd9e4f405e1c46c68cebd1a1204c5053"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-hfc3"
server_key: "b24a876e62a60bd583947083044fff83"
server_key_iv: "7915e502f10d7dabeeb9942598e428b1"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:34.654378 12338 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:28:34.668113 12355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.668386 12338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:34.668493 12338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "dd9e4f405e1c46c68cebd1a1204c5053"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-hfc3"
server_key: "b24a876e62a60bd583947083044fff83"
server_key_iv: "7915e502f10d7dabeeb9942598e428b1"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:34.668593 12338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 14:28:34 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:34.693225 12338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:34.695887 12338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:34.696058 12338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:34.696436 12338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:34.696938 12338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:34.697041 12338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.697129 12338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:34.697213 12338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.712236 12338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41179
I20260620 14:28:34.712847 12338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:28:34.714388 12468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41179 every 8 connection(s)
I20260620 14:28:34.719743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 12338
I20260620 14:28:34.719836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:28:34.720041 392 external_mini_cluster.cc:1428] Setting key 9860ad44488c21ffa9be5aa92e65d5a9
Jun 20 14:28:34 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:34.730159 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.715284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34153 (local address 127.0.98.2:51005)
0620 14:28:34.715629 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:34.715673 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:34.716510 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:34.716839 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:34.716843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:34.717218 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:34.717588 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.717595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.718615 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.718617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.720030 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.720034 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.720731 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.720732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.720906 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.724531 (+ 3625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:34.724541 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:34.726488 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:34.728473 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:34.728479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:34.728481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:34.728779 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:34.729033 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:34.729034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:34.729034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:34.729170 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:34.729475 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:34.729481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:34.729669 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 14:28:34.729779 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:28:34.729930 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":143,"threads_started":1}
I20260620 14:28:34.730883 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.715317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51005 (local address 127.0.98.62:34153)
0620 14:28:34.715368 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:34.715370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:34.715796 (+ 426us) server_negotiation.cc:407] Connection header received
0620 14:28:34.716662 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:34.716663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:34.716676 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:34.716755 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:34.717710 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.718511 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.720164 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.720529 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.720581 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.726579 (+ 5998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:34.726581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:34.726581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:34.726589 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:34.728376 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:34.728867 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:34.728869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:34.728869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:34.728940 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:34.729249 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:34.729250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:34.729251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:34.729391 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:34.729451 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:34.730708 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0620 14:28:34.730798 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:28:34.731442 12469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34153
I20260620 14:28:34.731556 12469 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:34.731752 12469 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
I20260620 14:28:34.732167 12120 ts_manager.cc:194] Registered new tserver with Master: dd9e4f405e1c46c68cebd1a1204c5053 (127.0.98.2:41179)
I20260620 14:28:34.732633 12120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51005
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:28:34.767660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34865/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34153
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:34.926244 12474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:34.926568 12474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:34.926623 12474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:34.928584 12474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:34.928680 12474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:34.928736 12474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:28:34.931257 12474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:34865/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:34.931561 12474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:34.931893 12474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:34.938746 12480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:34.939502 12481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:34.939706 12483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:34.940897 12474 server_base.cc:1034] running on GCE node
I20260620 14:28:34.941232 12474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:34.941511 12474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:34.942675 12474 hybrid_clock.cc:648] HybridClock initialized: now 1781965714942657 us; error 58 us; skew 500 ppm
Jun 20 14:28:34 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:34.946691 12474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:28:34.947410 12474 webserver.cc:460] Webserver started at http://127.0.98.3:41019/ using document root <none> and password file <none>
I20260620 14:28:34.947779 12474 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:34.947927 12474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:34.948179 12474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:28:34 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:34.968401 12474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "339bab90506e455187c739204faaeef9"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-hfc3"
server_key: "a4f356e80d68e5f484b2fafad0f43ca0"
server_key_iv: "d55994ef7a9d73348a54fa954783f055"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:34.969106 12474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "339bab90506e455187c739204faaeef9"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-hfc3"
server_key: "a4f356e80d68e5f484b2fafad0f43ca0"
server_key_iv: "d55994ef7a9d73348a54fa954783f055"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:34.972401 12474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:28:34.994508 12490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.997979 12474 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.000s
I20260620 14:28:34.998157 12474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "339bab90506e455187c739204faaeef9"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-hfc3"
server_key: "a4f356e80d68e5f484b2fafad0f43ca0"
server_key_iv: "d55994ef7a9d73348a54fa954783f055"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:34.998271 12474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:35.027733 12474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:35.030768 12474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:35.031061 12474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:35.031445 12474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:35.032071 12474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:35.032135 12474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:35.032168 12474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:35.032186 12474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:35.049600 12474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34941
I20260620 14:28:35.049727 12603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34941 every 8 connection(s)
I20260620 14:28:35.050169 12474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:28:35.059046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 12474
I20260620 14:28:35.059170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:28:35.059397 392 external_mini_cluster.cc:1428] Setting key 8ed97cc22742cfdeae98d0d0fade168a
Jun 20 14:28:35 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:35.067137 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:35.055172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41333 (local address 127.0.98.62:34153)
0620 14:28:35.055228 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:35.055230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:35.055659 (+ 429us) server_negotiation.cc:407] Connection header received
0620 14:28:35.056519 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:35.056520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:35.056534 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:35.056600 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:35.057724 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.058548 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.060382 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.060698 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.060763 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:35.064101 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:35.064103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:35.064103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:35.064110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:35.065612 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:35.066051 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:35.066053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:35.066053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:35.066113 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:35.066429 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:35.066430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:35.066430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:35.066550 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:35.066633 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:35.066954 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 14:28:35.067041 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:28:35.067368 12606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:35.055132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34153 (local address 127.0.98.3:41333)
0620 14:28:35.055441 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:35.055510 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:35.056383 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:35.056721 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:35.056733 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:35.057169 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:35.057597 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:35.057604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.058669 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.058670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:35.060276 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:35.060279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.060971 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.060972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:35.061251 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:35.062039 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:35.062050 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:35.063995 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:35.065719 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:35.065726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:35.065728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:35.065953 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:35.066185 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:35.066186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:35.066187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:35.066305 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:35.066625 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:35.066631 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:35.066814 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 14:28:35.066933 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:28:35.067093 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":167,"threads_started":1}
I20260620 14:28:35.067890 12604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34153
I20260620 14:28:35.067984 12604 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:35.068190 12604 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
I20260620 14:28:35.068619 12120 ts_manager.cc:194] Registered new tserver with Master: 339bab90506e455187c739204faaeef9 (127.0.98.3:34941)
I20260620 14:28:35.069252 12120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41333
I20260620 14:28:35.071556 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:28:35.072175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 12201
I20260620 14:28:35.079294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 12338
I20260620 14:28:35.084250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 12474
I20260620 14:28:35.088406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 12088
I20260620 14:28:35.094460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34153
--webserver_interface=127.0.98.62
--webserver_port=33823
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34153
--ranger_config_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34865/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34865/kms with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:35.175407 12607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:35.175627 12607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:35.175652 12607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:35.176709 12607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:35.176743 12607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:35.176757 12607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:35.176770 12607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:35.176782 12607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:35.178280 12607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34865/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34153
--ranger_config_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:35.178531 12607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:35.178714 12607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:35.183645 12615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:35.183645 12612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:35.183640 12613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:35.183889 12607 server_base.cc:1034] running on GCE node
I20260620 14:28:35.184144 12607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:35.184394 12607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:35.185602 12607 hybrid_clock.cc:648] HybridClock initialized: now 1781965715185589 us; error 31 us; skew 500 ppm
Jun 20 14:28:35 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:35.188143 12607 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:35.188623 12607 webserver.cc:460] Webserver started at http://127.0.98.62:33823/ using document root <none> and password file <none>
I20260620 14:28:35.188841 12607 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:35.188891 12607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:28:35 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:35.228485 12607 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:35.229270 12623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:35.229468 12607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:35.229597 12607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "693875695038486584e32e1731507c19"
format_stamp: "Formatted at 2026-06-20 14:28:33 on dist-test-slave-hfc3"
server_key: "7689103a550d57af9ae14838e7f50d68"
server_key_iv: "6d0f7280e5f478e1621fbb7ed2c8519d"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:35.229969 12607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:35.239207 12607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:35.241312 12607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:35.241475 12607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:35.245658 12607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34153
I20260620 14:28:35.245705 12675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34153 every 8 connection(s)
I20260620 14:28:35.246069 12607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:35.247462 12676 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:28:35.248204 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19: Bootstrap starting.
I20260620 14:28:35.250001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 12607
I20260620 14:28:35.250279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34865/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42419
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34153
--webserver_port=34999
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34865/kms with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:35.256862 12676 log.cc:824] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:35.258787 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:35.259133 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19: Bootstrap complete.
I20260620 14:28:35.261677 12676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693875695038486584e32e1731507c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34153 } }
I20260620 14:28:35.261912 12676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693875695038486584e32e1731507c19, State: Initialized, Role: FOLLOWER
I20260620 14:28:35.262099 12676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693875695038486584e32e1731507c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34153 } }
I20260620 14:28:35.262172 12676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:35.262223 12676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:35.262274 12676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:28:35.263361 12676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693875695038486584e32e1731507c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34153 } }
I20260620 14:28:35.263454 12676 leader_election.cc:302] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 693875695038486584e32e1731507c19; no voters:
I20260620 14:28:35.263691 12676 leader_election.cc:288] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:28:35.263739 12680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:28:35.263913 12680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [term 2 LEADER]: Becoming Leader. State: Replica: 693875695038486584e32e1731507c19, State: Running, Role: LEADER
I20260620 14:28:35.263972 12676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:35.264086 12680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693875695038486584e32e1731507c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34153 } }
I20260620 14:28:35.264344 12680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 693875695038486584e32e1731507c19. Latest consensus state: current_term: 2 leader_uuid: "693875695038486584e32e1731507c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693875695038486584e32e1731507c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34153 } } }
I20260620 14:28:35.264350 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "693875695038486584e32e1731507c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693875695038486584e32e1731507c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34153 } } }
I20260620 14:28:35.264423 12680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:35.264582 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:35.264845 12686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:35.265511 12686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:35.265683 12686 catalog_manager.cc:1259] Loaded cluster ID: 211ebb930d8f4485b25b3d4e66b49c7e
I20260620 14:28:35.265714 12686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:35.265923 12676 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:28:35.269373 12686 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:35.270021 12686 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19: Loaded TSK: 0
I20260620 14:28:35.270325 12686 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:28:35.360781 12678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:35.361198 12678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:35.361310 12678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:35.363189 12678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:35.363322 12678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:35.363421 12678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:28:35.365880 12678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34865/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34999
--tserver_master_addrs=127.0.98.62:34153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:35.366170 12678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:35.366550 12678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:35.373042 12714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:35.373168 12716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:35.373042 12713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:35.373762 12678 server_base.cc:1034] running on GCE node
I20260620 14:28:35.373982 12678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:35.374253 12678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:35.375403 12678 hybrid_clock.cc:648] HybridClock initialized: now 1781965715375377 us; error 41 us; skew 500 ppm
Jun 20 14:28:35 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:35.378837 12678 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:28:35.379357 12678 webserver.cc:460] Webserver started at http://127.0.98.1:34999/ using document root <none> and password file <none>
I20260620 14:28:35.379673 12678 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:35.379776 12678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:28:35 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:35.403786 12678 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:28:35.404752 12724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:35.404932 12678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:35.405009 12678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "592645c0b94b4337a1b16dbac8b77484"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-hfc3"
server_key: "4b2d8703e658e2a80a00946d0675a094"
server_key_iv: "cc1cda6cb6329c7ce4658c7f9002c163"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:35.405486 12678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:35.420251 12678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:35.422791 12678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:35.422971 12678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:35.423207 12678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:35.423514 12678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:35.423545 12678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:35.423569 12678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:35.423623 12678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:35.430007 12678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42419
I20260620 14:28:35.430092 12837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42419 every 8 connection(s)
I20260620 14:28:35.430507 12678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:28:35.430941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 12678
I20260620 14:28:35.431258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34865/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41179
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34153
--webserver_port=40881
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34865/kms with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:28:35 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:35.448910 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:35.431249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34153 (local address 127.0.98.1:50199)
0620 14:28:35.431520 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:35.431564 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:35.432195 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:35.433768 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:35.433772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:35.433977 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:35.434279 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:35.434286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.440061 (+ 5775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.440062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:35.441382 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:35.441385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.442308 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.442309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:35.442475 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:35.443185 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:35.443194 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:35.444649 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:35.446981 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:35.446987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:35.446990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:35.447288 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:35.447614 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:35.447615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:35.447615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:35.447750 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:35.448240 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:35.448246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:35.448404 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 14:28:35.448520 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:28:35.448674 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":85,"threads_started":1}
I20260620 14:28:35.449568 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:35.431368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50199 (local address 127.0.98.62:34153)
0620 14:28:35.433164 (+ 1796us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:35.433175 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:35.433220 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:28:35.433383 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:35.433391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:35.433449 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:35.433603 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:35.438445 (+ 4842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.439914 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.441466 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.442187 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.442315 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:35.444771 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:35.444776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:35.444776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:35.444788 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:35.446888 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:35.447382 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:35.447385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:35.447385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:35.447532 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:35.447830 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:35.447831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:35.447831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:35.448119 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:35.448211 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:35.449258 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0620 14:28:35.449375 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1728,"thread_start_us":78,"threads_started":1}
I20260620 14:28:35.450202 12838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34153
I20260620 14:28:35.450297 12838 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:35.450500 12838 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
W20260620 14:28:35.451359 12838 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34153 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:28:35.451427 12838 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:35.451488 12838 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
I20260620 14:28:35.451819 12838 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:35.451889 12838 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
W20260620 14:28:35.452203 12838 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:28:35.534811 12842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:35.535154 12842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:35.535225 12842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:35.536917 12842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:35.537006 12842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:35.537065 12842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:28:35.539281 12842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34865/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40881
--tserver_master_addrs=127.0.98.62:34153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:35.539529 12842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:35.539809 12842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:35.545540 12848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:35.545632 12850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:35.545528 12847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:35.546413 12842 server_base.cc:1034] running on GCE node
I20260620 14:28:35.546628 12842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:35.546928 12842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:35.548118 12842 hybrid_clock.cc:648] HybridClock initialized: now 1781965715548101 us; error 33 us; skew 500 ppm
Jun 20 14:28:35 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:35.551417 12842 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:28:35.551980 12842 webserver.cc:460] Webserver started at http://127.0.98.2:40881/ using document root <none> and password file <none>
I20260620 14:28:35.552241 12842 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:35.552326 12842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:28:35 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:35.572299 12842 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:28:35.574476 12858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:35.574698 12842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:28:35.574797 12842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "dd9e4f405e1c46c68cebd1a1204c5053"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-hfc3"
server_key: "b24a876e62a60bd583947083044fff83"
server_key_iv: "7915e502f10d7dabeeb9942598e428b1"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:35.575201 12842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:35.596046 12842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:35.598626 12842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:35.598856 12842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:35.599174 12842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:35.599568 12842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:35.599670 12842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:35.599747 12842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:35.599804 12842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:35.606137 12842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41179
I20260620 14:28:35.606703 12842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:28:35.609367 12971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41179 every 8 connection(s)
I20260620 14:28:35.615762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 12842
I20260620 14:28:35.616055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34865/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34941
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34153
--webserver_port=41019
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34865/kms with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:28:35 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:35.623595 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:35.607700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40745 (local address 127.0.98.62:34153)
0620 14:28:35.607748 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:35.607750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:35.608774 (+ 1024us) server_negotiation.cc:407] Connection header received
0620 14:28:35.609530 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:35.609532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:35.609547 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:35.609581 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:35.610498 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.611309 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.612814 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.613172 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.613251 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:35.618621 (+ 5370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:35.618622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:35.618623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:35.618632 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:35.621939 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:35.622470 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:35.622471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:35.622471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:35.622534 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:35.622866 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:35.622867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:35.622867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:35.623000 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:35.623046 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:35.623415 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 14:28:35.623504 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:28:35.624333 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:35.607879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34153 (local address 127.0.98.2:40745)
0620 14:28:35.608637 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:35.608681 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:35.609376 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:35.609803 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:35.609808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:35.610067 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:35.610389 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:35.610395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.611406 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.611407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:35.612707 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:35.612710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.613342 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.613344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:35.613499 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:35.614136 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:35.614147 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:35.618502 (+ 4355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:35.622039 (+ 3537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:35.622047 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:35.622049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:35.622382 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:35.622650 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:35.622650 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:35.622651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:35.622796 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:35.623077 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:35.623083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:35.623267 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 14:28:35.623932 (+ 665us) client_negotiation.cc:240] Negotiation successful
0620 14:28:35.624106 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":571,"threads_started":1}
I20260620 14:28:35.624998 12972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34153
I20260620 14:28:35.625098 12972 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:35.625311 12972 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
W20260620 14:28:35.625969 12972 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34153 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:28:35.626052 12972 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:35.626116 12972 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
I20260620 14:28:35.626470 12972 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:35.626559 12972 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
W20260620 14:28:35.626901 12972 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:28:35.737485 12975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:35.737857 12975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:35.737948 12975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:35.739804 12975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:35.739940 12975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:35.740052 12975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:28:35.742446 12975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34865/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41019
--tserver_master_addrs=127.0.98.62:34153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:35.742951 12975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:35.743319 12975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:35.749575 12981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:35.749953 12980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:35.754566 12983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:35.755879 12975 server_base.cc:1034] running on GCE node
I20260620 14:28:35.756108 12975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:35.756384 12975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:35.757664 12975 hybrid_clock.cc:648] HybridClock initialized: now 1781965715757639 us; error 48 us; skew 500 ppm
Jun 20 14:28:35 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:35.761135 12975 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:28:35.761885 12975 webserver.cc:460] Webserver started at http://127.0.98.3:41019/ using document root <none> and password file <none>
I20260620 14:28:35.762248 12975 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:35.762404 12975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:28:35 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:35.782300 12975 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:28:35.783217 12991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:35.783417 12975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:35.783497 12975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "339bab90506e455187c739204faaeef9"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-hfc3"
server_key: "a4f356e80d68e5f484b2fafad0f43ca0"
server_key_iv: "d55994ef7a9d73348a54fa954783f055"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:35.783921 12975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:35.806488 12975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:35.809063 12975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:35.809301 12975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:35.809595 12975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:35.809998 12975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:35.810083 12975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:35.810154 12975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:35.810221 12975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:35.816598 12975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34941
I20260620 14:28:35.817229 12975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:28:35.819797 13105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34941 every 8 connection(s)
I20260620 14:28:35.822806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 12975
Jun 20 14:28:35 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:35.833614 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:35.823436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.0.98.62:34153)
0620 14:28:35.823733 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:35.823735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:35.823912 (+ 177us) server_negotiation.cc:407] Connection header received
0620 14:28:35.824029 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:35.824030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:35.824045 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:35.824095 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:35.824986 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.825768 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.826945 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.827263 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.827319 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:35.829337 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:35.829338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:35.829339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:35.829347 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:35.831156 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:35.831652 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:35.831654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:35.831654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:35.831705 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:35.832247 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:35.832248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:35.832249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:35.832389 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:35.833339 (+ 950us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:35.833378 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 14:28:35.833456 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":63,"threads_started":1}
I20260620 14:28:35.835402 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:35.823416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45141 (local address 127.0.98.62:34153)
0620 14:28:35.823535 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:35.823537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:35.823549 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:28:35.824154 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:35.824155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:35.824167 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:35.824229 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:35.825089 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.825858 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.827470 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.827737 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.827784 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:35.831430 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:35.831431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:35.831432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:35.831438 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:35.833161 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:35.833973 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:35.833973 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:35.833974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:35.834035 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:35.834598 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:35.834599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:35.834600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:35.834736 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:35.834784 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:35.835223 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 14:28:35.835309 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 14:28:35.836055 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:35.823067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34153 (local address 127.0.98.3:45141)
0620 14:28:35.823302 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:35.823362 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:35.824008 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:35.824442 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:35.824446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:35.824694 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:35.824973 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:35.824979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.825980 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.825981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:35.827362 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:35.827365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:35.827893 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:35.827894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:35.828048 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:35.828603 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:35.828613 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:35.831334 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:35.833262 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:35.833271 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:35.833273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:35.833590 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:35.834205 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:35.834206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:35.834206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:35.834472 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:35.834836 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:35.834843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:35.835035 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 14:28:35.835625 (+ 590us) client_negotiation.cc:240] Negotiation successful
0620 14:28:35.835807 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":125,"threads_started":1}
I20260620 14:28:35.836566 13106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34153
I20260620 14:28:35.836700 13106 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:35.836911 13106 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
W20260620 14:28:35.837663 13106 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34153 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:28:35.837752 13106 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:35.837813 13106 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
I20260620 14:28:35.838202 13106 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:35.838281 13106 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
W20260620 14:28:35.838639 13106 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:28:36 dist-test-slave-hfc3 krb5kdc[11644](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:36.452384 12838 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:36.452548 12838 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
I20260620 14:28:36.453033 12640 ts_manager.cc:194] Registered new tserver with Master: 592645c0b94b4337a1b16dbac8b77484 (127.0.98.1:42419)
I20260620 14:28:36.454069 12640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50199
I20260620 14:28:36.627151 12972 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:36.627471 12972 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
I20260620 14:28:36.627956 12640 ts_manager.cc:194] Registered new tserver with Master: dd9e4f405e1c46c68cebd1a1204c5053 (127.0.98.2:41179)
I20260620 14:28:36.628374 12640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40745
Jun 20 14:28:36 dist-test-slave-hfc3 krb5kdc[11644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:36.838819 13106 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:36.838984 13106 heartbeater.cc:511] Master 127.0.98.62:34153 requested a full tablet report, sending...
I20260620 14:28:36.839331 12640 ts_manager.cc:194] Registered new tserver with Master: 339bab90506e455187c739204faaeef9 (127.0.98.3:34941)
I20260620 14:28:36.839860 12640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45141
I20260620 14:28:36.841535 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:28:36.856117 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:36.843919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.0.98.62:34153)
0620 14:28:36.844935 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:36.844937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:36.845095 (+ 158us) server_negotiation.cc:407] Connection header received
0620 14:28:36.845202 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:36.845203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:36.845214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:36.845290 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:36.846189 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.847034 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.848111 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.848428 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.848469 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:36.852809 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:36.852811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:36.852812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:36.852820 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:36.854768 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:36.855174 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:36.855175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:36.855176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:36.855233 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:36.855437 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:36.855437 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:36.855438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:36.855585 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:36.855685 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:36.855928 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 14:28:36.856021 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":57,"threads_started":1}
I20260620 14:28:36.858822 12640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:28:36.861119 12640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:28:36.868528 13133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:36.864182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41179 (local address 127.0.0.1:45130)
0620 14:28:36.865008 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:36.865018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:36.865091 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:36.865455 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:36.865458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:36.865460 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:36.865533 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:36.865535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.866479 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.866480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:36.867468 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:36.867470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.868196 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.868197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:36.868284 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:36.868312 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:28:36.868395 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:28:36.868448 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":107,"threads_started":1}
I20260620 14:28:36.868549 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:36.864734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.0.98.2:41179)
0620 14:28:36.865197 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:36.865202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:36.865212 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:28:36.865269 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:36.865270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:36.865286 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:36.865416 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:36.865630 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.866410 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.867555 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.868102 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.868196 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:36.868326 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:36.868409 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 14:28:36.868448 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":59,"threads_started":1}
I20260620 14:28:36.869735 12906 tablet_service.cc:1467] Processing CreateTablet for tablet 7d675c7abd534598afa47750e976ef8d (DEFAULT_TABLE table=test-table [id=ba971a6967f942a984b7bca3a893ca1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:36.870107 12906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d675c7abd534598afa47750e976ef8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:36.873471 13139 tablet_bootstrap.cc:492] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053: Bootstrap starting.
I20260620 14:28:36.874528 13139 tablet_bootstrap.cc:654] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:36.875020 13139 log.cc:824] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:36.875982 13139 tablet_bootstrap.cc:492] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053: No bootstrap required, opened a new log
I20260620 14:28:36.876150 13139 ts_tablet_manager.cc:1397] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:28:36.880620 13139 raft_consensus.cc:348] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592645c0b94b4337a1b16dbac8b77484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42419 } } peers { permanent_uuid: "dd9e4f405e1c46c68cebd1a1204c5053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "339bab90506e455187c739204faaeef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34941 } }
I20260620 14:28:36.880923 13139 raft_consensus.cc:374] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:36.881021 13133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:36.864416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34941 (local address 127.0.0.1:34986)
0620 14:28:36.868935 (+ 4519us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:36.868946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:36.868992 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:36.875975 (+ 6983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:36.875976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:36.875978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:36.877135 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:36.877136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.878557 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.878558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:36.879748 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:36.879751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.880789 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.880790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:36.880855 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:36.880860 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:28:36.880888 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 14:28:36.880928 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4504,"mutex_wait_us":1115,"thread_start_us":95,"threads_started":1}
I20260620 14:28:36.881037 13139 raft_consensus.cc:729] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd9e4f405e1c46c68cebd1a1204c5053, State: Initialized, Role: FOLLOWER
I20260620 14:28:36.881199 13139 consensus_queue.cc:260] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592645c0b94b4337a1b16dbac8b77484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42419 } } peers { permanent_uuid: "dd9e4f405e1c46c68cebd1a1204c5053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "339bab90506e455187c739204faaeef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34941 } }
I20260620 14:28:36.881613 13139 ts_tablet_manager.cc:1428] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260620 14:28:36.881747 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:36.864527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.0.98.3:34941)
0620 14:28:36.875740 (+ 11213us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:36.875744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:36.875755 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:28:36.875789 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:36.875791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:36.875807 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:36.876009 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:36.877218 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.878457 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.879848 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.880686 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.881447 (+ 761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:36.881591 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:36.881620 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 14:28:36.881661 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11169,"thread_start_us":61,"threads_started":1}
I20260620 14:28:36.881809 12972 heartbeater.cc:503] Master 127.0.98.62:34153 was elected leader, sending a full tablet report...
I20260620 14:28:36.882588 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 7d675c7abd534598afa47750e976ef8d (DEFAULT_TABLE table=test-table [id=ba971a6967f942a984b7bca3a893ca1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:36.882938 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d675c7abd534598afa47750e976ef8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:36.883445 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:36.864141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42419 (local address 127.0.0.1:47232)
0620 14:28:36.864638 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:36.864680 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:36.864783 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:36.874397 (+ 9614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:36.874398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:36.874401 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:36.874454 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:36.874456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.875877 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.875878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:36.877033 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:36.877036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.883204 (+ 6168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.883205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:36.883272 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:36.883277 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:28:36.883309 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 14:28:36.883355 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":128,"threads_started":1}
I20260620 14:28:36.883841 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:36.864556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.0.98.1:42419)
0620 14:28:36.874037 (+ 9481us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:36.874043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:36.874056 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:28:36.874093 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:36.874095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:36.874177 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:36.874527 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:36.874628 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.875774 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.882205 (+ 6431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.883056 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.883603 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:36.883672 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:36.883701 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 14:28:36.883741 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9437,"thread_start_us":81,"threads_started":1}
I20260620 14:28:36.884650 12772 tablet_service.cc:1467] Processing CreateTablet for tablet 7d675c7abd534598afa47750e976ef8d (DEFAULT_TABLE table=test-table [id=ba971a6967f942a984b7bca3a893ca1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:36.884967 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d675c7abd534598afa47750e976ef8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:36.887096 13142 tablet_bootstrap.cc:492] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9: Bootstrap starting.
I20260620 14:28:36.888018 13142 tablet_bootstrap.cc:654] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:36.888595 13142 log.cc:824] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:36.889899 13144 tablet_bootstrap.cc:492] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484: Bootstrap starting.
I20260620 14:28:36.890908 13142 tablet_bootstrap.cc:492] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9: No bootstrap required, opened a new log
I20260620 14:28:36.891006 13142 ts_tablet_manager.cc:1397] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:28:36.893755 13142 raft_consensus.cc:348] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592645c0b94b4337a1b16dbac8b77484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42419 } } peers { permanent_uuid: "dd9e4f405e1c46c68cebd1a1204c5053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "339bab90506e455187c739204faaeef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34941 } }
I20260620 14:28:36.893922 13142 raft_consensus.cc:374] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:36.893965 13142 raft_consensus.cc:729] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339bab90506e455187c739204faaeef9, State: Initialized, Role: FOLLOWER
I20260620 14:28:36.894121 13142 consensus_queue.cc:260] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592645c0b94b4337a1b16dbac8b77484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42419 } } peers { permanent_uuid: "dd9e4f405e1c46c68cebd1a1204c5053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "339bab90506e455187c739204faaeef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34941 } }
I20260620 14:28:36.894485 13106 heartbeater.cc:503] Master 127.0.98.62:34153 was elected leader, sending a full tablet report...
I20260620 14:28:36.894477 13142 ts_tablet_manager.cc:1428] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:28:36.895467 13144 tablet_bootstrap.cc:654] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:36.896010 13144 log.cc:824] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:36.896956 13144 tablet_bootstrap.cc:492] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484: No bootstrap required, opened a new log
I20260620 14:28:36.897045 13144 ts_tablet_manager.cc:1397] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260620 14:28:36.900167 13144 raft_consensus.cc:348] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592645c0b94b4337a1b16dbac8b77484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42419 } } peers { permanent_uuid: "dd9e4f405e1c46c68cebd1a1204c5053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "339bab90506e455187c739204faaeef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34941 } }
I20260620 14:28:36.900329 13144 raft_consensus.cc:374] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:36.900405 13144 raft_consensus.cc:729] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592645c0b94b4337a1b16dbac8b77484, State: Initialized, Role: FOLLOWER
I20260620 14:28:36.900554 13144 consensus_queue.cc:260] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592645c0b94b4337a1b16dbac8b77484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42419 } } peers { permanent_uuid: "dd9e4f405e1c46c68cebd1a1204c5053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "339bab90506e455187c739204faaeef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34941 } }
I20260620 14:28:36.901175 12838 heartbeater.cc:503] Master 127.0.98.62:34153 was elected leader, sending a full tablet report...
I20260620 14:28:36.901511 13144 ts_tablet_manager.cc:1428] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260620 14:28:36.934095 12839 tablet.cc:2369] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:28:36.974061 13145 raft_consensus.cc:482] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:36.974488 13145 raft_consensus.cc:504] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592645c0b94b4337a1b16dbac8b77484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42419 } } peers { permanent_uuid: "dd9e4f405e1c46c68cebd1a1204c5053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "339bab90506e455187c739204faaeef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34941 } }
I20260620 14:28:36.975206 13145 leader_election.cc:288] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 592645c0b94b4337a1b16dbac8b77484 (127.0.98.1:42419), dd9e4f405e1c46c68cebd1a1204c5053 (127.0.98.2:41179)
I20260620 14:28:36.979686 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:36.975659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44723 (local address 127.0.98.2:41179)
0620 14:28:36.975724 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:36.975726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:36.975955 (+ 229us) server_negotiation.cc:407] Connection header received
0620 14:28:36.976058 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:36.976059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:36.976071 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:36.976099 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:36.976307 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.977043 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.978633 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.979269 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.979321 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:36.979332 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:36.979568 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 14:28:36.979613 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:28:36.979938 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:36.975631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45755 (local address 127.0.98.1:42419)
0620 14:28:36.975681 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:36.975683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:36.975851 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:28:36.975972 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:36.975974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:36.975987 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:36.976018 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:36.976327 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.977026 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.978641 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.979284 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.979336 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:36.979345 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:36.979829 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 14:28:36.979871 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:28:36.980188 13148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:36.975392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42419 (local address 127.0.98.3:45755)
0620 14:28:36.975745 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:36.975756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:36.975884 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:36.976091 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:36.976092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:36.976094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:36.976214 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:36.976217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.977133 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.977134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:36.978531 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:36.978534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.979670 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.979671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:36.979749 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:36.979752 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:28:36.980065 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 14:28:36.980110 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":71,"thread_start_us":125,"threads_started":1}
I20260620 14:28:36.980430 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:36.975433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41179 (local address 127.0.98.3:44723)
0620 14:28:36.975855 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:36.975865 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:36.975983 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:36.976161 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:36.976162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:36.976163 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:36.976235 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:36.976237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.977215 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.977216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:36.978533 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:36.978535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:36.979414 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:36.979415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:36.979482 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:36.979486 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:28:36.980208 (+ 722us) client_negotiation.cc:240] Negotiation successful
0620 14:28:36.980278 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":115,"threads_started":1}
I20260620 14:28:36.980772 12792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d675c7abd534598afa47750e976ef8d" candidate_uuid: "339bab90506e455187c739204faaeef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592645c0b94b4337a1b16dbac8b77484" is_pre_election: true
I20260620 14:28:36.980939 12792 raft_consensus.cc:2454] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 339bab90506e455187c739204faaeef9 in term 0.
I20260620 14:28:36.980953 12924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d675c7abd534598afa47750e976ef8d" candidate_uuid: "339bab90506e455187c739204faaeef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd9e4f405e1c46c68cebd1a1204c5053" is_pre_election: true
I20260620 14:28:36.981096 12924 raft_consensus.cc:2454] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 339bab90506e455187c739204faaeef9 in term 0.
I20260620 14:28:36.981171 12993 leader_election.cc:302] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 339bab90506e455187c739204faaeef9, 592645c0b94b4337a1b16dbac8b77484; no voters:
I20260620 14:28:36.981361 13145 raft_consensus.cc:2781] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:28:36.981411 13145 raft_consensus.cc:482] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:28:36.981436 13145 raft_consensus.cc:3037] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:36.982426 13145 raft_consensus.cc:504] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592645c0b94b4337a1b16dbac8b77484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42419 } } peers { permanent_uuid: "dd9e4f405e1c46c68cebd1a1204c5053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "339bab90506e455187c739204faaeef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34941 } }
I20260620 14:28:36.982569 13145 leader_election.cc:288] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [CANDIDATE]: Term 1 election: Requested vote from peers 592645c0b94b4337a1b16dbac8b77484 (127.0.98.1:42419), dd9e4f405e1c46c68cebd1a1204c5053 (127.0.98.2:41179)
I20260620 14:28:36.982749 12792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d675c7abd534598afa47750e976ef8d" candidate_uuid: "339bab90506e455187c739204faaeef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592645c0b94b4337a1b16dbac8b77484"
I20260620 14:28:36.982806 12924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d675c7abd534598afa47750e976ef8d" candidate_uuid: "339bab90506e455187c739204faaeef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd9e4f405e1c46c68cebd1a1204c5053"
I20260620 14:28:36.982827 12792 raft_consensus.cc:3037] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:36.982887 12924 raft_consensus.cc:3037] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:36.983762 12792 raft_consensus.cc:2454] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 339bab90506e455187c739204faaeef9 in term 1.
I20260620 14:28:36.983814 12924 raft_consensus.cc:2454] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 339bab90506e455187c739204faaeef9 in term 1.
I20260620 14:28:36.983990 12994 leader_election.cc:302] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 339bab90506e455187c739204faaeef9, dd9e4f405e1c46c68cebd1a1204c5053; no voters:
I20260620 14:28:36.984122 13145 raft_consensus.cc:2781] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:36.984457 13145 raft_consensus.cc:686] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 1 LEADER]: Becoming Leader. State: Replica: 339bab90506e455187c739204faaeef9, State: Running, Role: LEADER
I20260620 14:28:36.984566 13145 consensus_queue.cc:237] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592645c0b94b4337a1b16dbac8b77484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42419 } } peers { permanent_uuid: "dd9e4f405e1c46c68cebd1a1204c5053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "339bab90506e455187c739204faaeef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34941 } }
I20260620 14:28:36.985405 12640 catalog_manager.cc:5495] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 339bab90506e455187c739204faaeef9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "339bab90506e455187c739204faaeef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592645c0b94b4337a1b16dbac8b77484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd9e4f405e1c46c68cebd1a1204c5053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "339bab90506e455187c739204faaeef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34941 } health_report { overall_health: HEALTHY } } }
I20260620 14:28:37.014230 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:37.010931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.0.98.3:34941)
0620 14:28:37.011009 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:37.011011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:37.011021 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:28:37.011067 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:37.011069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:37.011095 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:37.011184 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:28:37.011360 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:37.012098 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:37.013212 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:37.013536 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:37.013612 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:37.013826 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:28:37.013971 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:28:37.014055 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:37.014120 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 14:28:37.014143 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:28:37.016459 12924 raft_consensus.cc:1261] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053 [term 1 FOLLOWER]: Refusing update from remote peer 339bab90506e455187c739204faaeef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:37.016554 12792 raft_consensus.cc:1261] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484 [term 1 FOLLOWER]: Refusing update from remote peer 339bab90506e455187c739204faaeef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:37.016968 13150 consensus_queue.cc:1035] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "592645c0b94b4337a1b16dbac8b77484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:37.017141 13150 consensus_queue.cc:1035] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd9e4f405e1c46c68cebd1a1204c5053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:37.019979 13153 mvcc.cc:205] Tried to move back new op lower bound from 7298931576896634880 to 7298931576769335296. Current Snapshot: MvccSnapshot[applied={T|T < 7298931576896634880}]
I20260620 14:28:37.021417 13155 mvcc.cc:205] Tried to move back new op lower bound from 7298931576896634880 to 7298931576769335296. Current Snapshot: MvccSnapshot[applied={T|T < 7298931576896634880}]
I20260620 14:28:37.022505 12639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:37.022599 12639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:37.023499 12639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 693875695038486584e32e1731507c19: Sending DeleteTablet for 3 replicas of tablet 7d675c7abd534598afa47750e976ef8d
I20260620 14:28:37.023947 12906 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d675c7abd534598afa47750e976ef8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45130
I20260620 14:28:37.023985 12772 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d675c7abd534598afa47750e976ef8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47232
I20260620 14:28:37.024226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 12678
I20260620 14:28:37.024263 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d675c7abd534598afa47750e976ef8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34986
I20260620 14:28:37.024304 13161 tablet_replica.cc:330] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484: stopping tablet replica
I20260620 14:28:37.024389 13161 raft_consensus.cc:2229] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:37.024468 13161 raft_consensus.cc:2258] T 7d675c7abd534598afa47750e976ef8d P 592645c0b94b4337a1b16dbac8b77484 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:37.028663 13154 mvcc.cc:205] Tried to move back new op lower bound from 7298931576896634880 to 7298931576769335296. Current Snapshot: MvccSnapshot[applied={T|T < 7298931576896634880}]
I20260620 14:28:37.028859 13163 tablet_replica.cc:330] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9: stopping tablet replica
I20260620 14:28:37.028998 13163 raft_consensus.cc:2229] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:28:37.029114 13163 raft_consensus.cc:2258] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:37.029277 13162 tablet_replica.cc:330] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053: stopping tablet replica
I20260620 14:28:37.029374 13162 raft_consensus.cc:2229] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:37.029479 13162 raft_consensus.cc:2258] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:37.029922 13163 ts_tablet_manager.cc:1905] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:37.029930 13162 ts_tablet_manager.cc:1905] T 7d675c7abd534598afa47750e976ef8d P dd9e4f405e1c46c68cebd1a1204c5053: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:28:37.032393 12625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42419 (error 108)
I20260620 14:28:37.032948 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:37.032822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42419 (local address 127.0.0.1:47238)
0620 14:28:37.032876 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:37.032912 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:28:37.033079 12625 catalog_manager.cc:4584] TS 592645c0b94b4337a1b16dbac8b77484 (127.0.98.1:42419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d675c7abd534598afa47750e976ef8d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42419: connect: Connection refused (error 111)
I20260620 14:28:37.033579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 12842
I20260620 14:28:37.046612 13163 ts_tablet_manager.cc:1918] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:37.046828 13163 log.cc:1196] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/7d675c7abd534598afa47750e976ef8d
I20260620 14:28:37.047206 13163 ts_tablet_manager.cc:1939] T 7d675c7abd534598afa47750e976ef8d P 339bab90506e455187c739204faaeef9: Deleting consensus metadata
I20260620 14:28:37.047814 12626 catalog_manager.cc:4841] TS 339bab90506e455187c739204faaeef9 (127.0.98.3:34941): tablet 7d675c7abd534598afa47750e976ef8d (table test-table [id=ba971a6967f942a984b7bca3a893ca1d]) successfully deleted
W20260620 14:28:37.048487 13134 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:28:37.048172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41179 (local address 127.0.0.1:45140)
0620 14:28:37.048228 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:37.048239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:37.048278 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:37.048415 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41179: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:28:37.049140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 12975
I20260620 14:28:37.055419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 12607
I20260620 14:28:37.061436 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:28:37.395953 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:28:37.396050 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:28:37.740460 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:28:37.740 UTC [11674] LOG: received smart shutdown request
2026-06-20 14:28:37.745 UTC [11674] LOG: background worker "logical replication launcher" (PID 11683) exited with exit code 1
2026-06-20 14:28:37.745 UTC [11678] LOG: shutting down
2026-06-20 14:28:37.921 UTC [11674] LOG: database system is shut down
2026-06-20T14:28:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48943 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:28:38 dist-test-slave-hfc3 krb5kdc[13174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:28:38 dist-test-slave-hfc3 krb5kdc[13174](info): set up 2 sockets
Jun 20 14:28:38 dist-test-slave-hfc3 krb5kdc[13174](info): commencing operation
krb5kdc: starting...
W20260620 14:28:39.074090 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:28:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:28:39.196741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38591
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38591,127.0.98.61:34191,127.0.98.60:36121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:39.277771 13187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:39.278048 13187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:39.278087 13187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:39.279300 13187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:39.279387 13187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:39.279414 13187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:39.279436 13187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:39.279464 13187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:39.281055 13187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38591,127.0.98.61:34191,127.0.98.60:36121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:39.281353 13187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:39.281596 13187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:39.286649 13195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:39.286684 13192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:39.286686 13193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:39.286949 13187 server_base.cc:1034] running on GCE node
I20260620 14:28:39.287223 13187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:39.287524 13187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:39.288697 13187 hybrid_clock.cc:648] HybridClock initialized: now 1781965719288672 us; error 44 us; skew 500 ppm
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:39.291445 13187 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:39.291970 13187 webserver.cc:460] Webserver started at http://127.0.98.62:36341/ using document root <none> and password file <none>
I20260620 14:28:39.292271 13187 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:39.292339 13187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:39.292488 13187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:39.293746 13187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "a71a395bb6a8432787a2ce53a3837ab0"
format_stamp: "Formatted at 2026-06-20 14:28:39 on dist-test-slave-hfc3"
server_key: "ced29e90e19f76140e144ec0315e203a"
server_key_iv: "8f1ab622cbd1c84e9a1922a989821178"
server_key_version: "encryptionkey@0"
I20260620 14:28:39.294137 13187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "a71a395bb6a8432787a2ce53a3837ab0"
format_stamp: "Formatted at 2026-06-20 14:28:39 on dist-test-slave-hfc3"
server_key: "ced29e90e19f76140e144ec0315e203a"
server_key_iv: "8f1ab622cbd1c84e9a1922a989821178"
server_key_version: "encryptionkey@0"
I20260620 14:28:39.295586 13187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:28:39.296396 13202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:39.296586 13187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:39.296695 13187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "a71a395bb6a8432787a2ce53a3837ab0"
format_stamp: "Formatted at 2026-06-20 14:28:39 on dist-test-slave-hfc3"
server_key: "ced29e90e19f76140e144ec0315e203a"
server_key_iv: "8f1ab622cbd1c84e9a1922a989821178"
server_key_version: "encryptionkey@0"
I20260620 14:28:39.296782 13187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:39.316071 13187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:39.318713 13187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:39.318892 13187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:39.323297 13187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38591
I20260620 14:28:39.323338 13254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38591 every 8 connection(s)
I20260620 14:28:39.323724 13187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:39.324594 13255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:39.326256 13255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:39.331542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 13187
I20260620 14:28:39.331669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:28:39.331835 392 external_mini_cluster.cc:1428] Setting key e4f8b4bacbb55c3e243e64ea1b740a10
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:39.337306 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.326999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38591 (local address 127.0.0.1:53898)
0620 14:28:39.327151 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.327187 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:39.327702 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:39.328009 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:39.328011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:39.328303 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:39.328458 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.328462 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.329504 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.329505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.330206 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.330208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.330782 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.330783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.330849 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.331434 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:39.331442 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:39.333286 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:39.335753 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.335761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.335763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.335992 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.336302 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.336303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.336303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.336386 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.336852 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:39.336857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:39.336918 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 14:28:39.336993 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:28:39.337123 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260620 14:28:39.337393 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.327148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.0.98.62:38591)
0620 14:28:39.327260 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.327265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.327305 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:28:39.327883 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.327885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.327901 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.328003 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.328565 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.329383 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.330315 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.330673 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.330822 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.333393 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.333403 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.333403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.333411 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.335658 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.336100 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.336104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.336104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.336222 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.336458 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.336459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.336459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.336777 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.336884 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.337057 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.337212 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":85,"thread_start_us":41,"threads_started":1}
I20260620 14:28:39.338202 13255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:39.338614 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.338468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34191 (local address 127.0.0.1:51746)
0620 14:28:39.338527 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.338584 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:28:39.338836 13204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34191: connect: Connection refused (error 111)
I20260620 14:28:39.339409 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.339284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34191 (local address 127.0.0.1:51762)
0620 14:28:39.339343 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.339377 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:28:39.339558 13255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34191: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34191: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:28:39.357184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34191
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38591,127.0.98.61:34191,127.0.98.60:36121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:39.388913 13255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } attempt: 1
I20260620 14:28:39.389556 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.389405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34191 (local address 127.0.0.1:51778)
0620 14:28:39.389478 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.389526 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:28:39.389966 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.389870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34191 (local address 127.0.0.1:51790)
0620 14:28:39.389925 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.389949 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 14:28:39.390089 13255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34191: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34191: connect: Connection refused (error 111)
W20260620 14:28:39.437973 13262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:39.438184 13262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:39.438207 13262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:39.439276 13262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:39.439311 13262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:39.439324 13262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:39.439337 13262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:39.439349 13262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:39.440789 13262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38591,127.0.98.61:34191,127.0.98.60:36121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:39.440995 13262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:39.441161 13262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:39.445797 13270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:39.445834 13267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:39.445924 13268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:39.446197 13262 server_base.cc:1034] running on GCE node
I20260620 14:28:39.446473 13262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:39.446818 13262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:39.447962 13262 hybrid_clock.cc:648] HybridClock initialized: now 1781965719447950 us; error 28 us; skew 500 ppm
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:39.450554 13262 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:28:39.450995 13262 webserver.cc:460] Webserver started at http://127.0.98.61:41201/ using document root <none> and password file <none>
I20260620 14:28:39.451232 13262 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:39.451306 13262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:39.451465 13262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:39.452531 13262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/data/instance:
uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3"
format_stamp: "Formatted at 2026-06-20 14:28:39 on dist-test-slave-hfc3"
server_key: "0e3db67b895cd070564296fcabbc58ec"
server_key_iv: "57cae8fc190b599a4a871fd9c41a8989"
server_key_version: "encryptionkey@0"
I20260620 14:28:39.452908 13262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/wal/instance:
uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3"
format_stamp: "Formatted at 2026-06-20 14:28:39 on dist-test-slave-hfc3"
server_key: "0e3db67b895cd070564296fcabbc58ec"
server_key_iv: "57cae8fc190b599a4a871fd9c41a8989"
server_key_version: "encryptionkey@0"
I20260620 14:28:39.454205 13262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:28:39.455020 13277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:39.455188 13262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:28:39.455291 13262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/wal
uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3"
format_stamp: "Formatted at 2026-06-20 14:28:39 on dist-test-slave-hfc3"
server_key: "0e3db67b895cd070564296fcabbc58ec"
server_key_iv: "57cae8fc190b599a4a871fd9c41a8989"
server_key_version: "encryptionkey@0"
I20260620 14:28:39.455372 13262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:39.458249 13255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } attempt: 2
I20260620 14:28:39.458691 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.458533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34191 (local address 127.0.0.1:51794)
0620 14:28:39.458613 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.458653 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:28:39.459084 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.458985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34191 (local address 127.0.0.1:51804)
0620 14:28:39.459032 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.459059 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 14:28:39.459215 13255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34191: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34191: connect: Connection refused (error 111)
I20260620 14:28:39.477114 13262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:39.479744 13262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:39.479938 13262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:39.484109 13262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34191
I20260620 14:28:39.484200 13329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34191 every 8 connection(s)
I20260620 14:28:39.484532 13262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:28:39.485437 13330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:39.487047 13330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:39.492197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 13262
I20260620 14:28:39.492275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-1/wal/instance
I20260620 14:28:39.492484 392 external_mini_cluster.cc:1428] Setting key 24179c51a376fa5a7c68bcd6819672c6
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:39.496114 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.487761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.0.98.62:38591)
0620 14:28:39.487808 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.487810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.488056 (+ 246us) server_negotiation.cc:407] Connection header received
0620 14:28:39.488492 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.488493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.488506 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.488560 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.489147 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.489637 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.490627 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.490823 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.490877 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.493144 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.493146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.493147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.493155 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.494679 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.495111 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.495112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.495113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.495176 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.495441 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.495441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.495442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.495557 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.495635 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.495937 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.496023 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:28:39.496330 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.487747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38591 (local address 127.0.0.1:49850)
0620 14:28:39.487930 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.487969 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:39.488409 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:39.488638 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:39.488641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:39.488881 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:39.489076 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.489080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.489718 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.489719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.490558 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.490560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.490897 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.490898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.490993 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.491531 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:39.491539 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:39.493038 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:39.494781 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.494789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.494791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.495030 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.495245 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.495246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.495247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.495364 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.495627 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:39.495634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:39.495782 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 14:28:39.495913 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:28:39.496077 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260620 14:28:39.496901 13330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:28:39.506776 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.497164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34191 (local address 127.0.0.1:48268)
0620 14:28:39.497215 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.497226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:39.497266 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:39.497588 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:39.497588 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:39.497702 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:39.497750 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.497752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.499104 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.499104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.499698 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.499700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.500409 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.500414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.500488 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.500935 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:39.500941 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:39.503844 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:39.505660 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.505661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.505661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.505828 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.506061 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.506062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.506063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.506116 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.506440 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:39.506440 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:39.506482 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:28:39.506554 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:28:39.506660 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:28:39.506846 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.497221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.98.61:34191)
0620 14:28:39.497407 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.497412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.497438 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:28:39.497462 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.497465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.497480 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.497568 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.497823 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.499019 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.499790 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.500322 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.500401 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.503951 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.503961 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.503961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.503968 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.505550 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.505932 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.505935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.505936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.505991 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.506187 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.506188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.506188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.506381 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.506463 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.506585 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.506730 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260620 14:28:39.507429 13330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:39.507844 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.507681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39586)
0620 14:28:39.507739 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.507804 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:28:39.508018 13281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
I20260620 14:28:39.508463 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.508338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39596)
0620 14:28:39.508399 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.508432 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 14:28:39.508610 13330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:28:39.519390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36121
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38591,127.0.98.61:34191,127.0.98.60:36121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:39.550417 13255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } attempt: 3
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:28:39.557858 13330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } attempt: 1
I20260620 14:28:39.558375 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.558185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39598)
0620 14:28:39.558252 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.558291 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:28:39.558434 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.550827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34191 (local address 127.0.0.1:48274)
0620 14:28:39.550913 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.550925 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:39.550973 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:39.551129 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:39.551130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:39.551317 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:39.551395 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.551397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.552087 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.552088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.552878 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.552880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.553256 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.553257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.553342 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.553809 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:39.553815 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:39.555244 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:39.556917 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.556919 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.556919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.557178 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.557386 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.557387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.557388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.557471 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.557902 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:39.557903 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:39.557977 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 14:28:39.558053 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:28:39.558307 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 14:28:39.558517 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.550899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.0.98.61:34191)
0620 14:28:39.550968 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.550970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.550984 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:28:39.551025 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.551026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.551036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.551099 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.551473 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.551995 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.552972 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.553168 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.553218 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.555336 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.555337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.555338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.555345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.556813 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.557261 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.557262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.557263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.557313 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.557550 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.557550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.557551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.557804 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.557884 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.558169 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.558259 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:28:39.559147 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.558951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39600)
0620 14:28:39.559080 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.559104 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:28:39.559235 13330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
I20260620 14:28:39.559221 13255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:39.559690 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.559532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39614)
0620 14:28:39.559602 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.559635 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 14:28:39.560089 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.559980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39622)
0620 14:28:39.560044 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.560070 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:28:39.560252 13255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
I20260620 14:28:39.591426 13255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } attempt: 1
I20260620 14:28:39.592008 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.591851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39630)
0620 14:28:39.591926 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.591973 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:28:39.592425 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.592310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39644)
0620 14:28:39.592370 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.592401 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 14:28:39.592578 13255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
W20260620 14:28:39.605067 13337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:39.605309 13337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:39.605347 13337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:39.606571 13337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:39.606637 13337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:39.606663 13337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:39.606696 13337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:39.606721 13337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:39.608238 13337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38591,127.0.98.61:34191,127.0.98.60:36121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:39.608520 13337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:39.608748 13337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:39.614365 13343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:39.614353 13345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:39.614643 13342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:39.614815 13337 server_base.cc:1034] running on GCE node
I20260620 14:28:39.615120 13337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:39.615465 13337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:39.616634 13337 hybrid_clock.cc:648] HybridClock initialized: now 1781965719616615 us; error 34 us; skew 500 ppm
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:39.619994 13337 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:28:39.620584 13337 webserver.cc:460] Webserver started at http://127.0.98.60:35005/ using document root <none> and password file <none>
I20260620 14:28:39.620848 13337 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:39.620906 13337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:39.620998 13337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:39.622037 13337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/data/instance:
uuid: "fed9c0217989495697d98fbb361fd67c"
format_stamp: "Formatted at 2026-06-20 14:28:39 on dist-test-slave-hfc3"
server_key: "84a785cf339a9d7380ea90996d4766f7"
server_key_iv: "1ad3a84caf1a980092752c3502430332"
server_key_version: "encryptionkey@0"
I20260620 14:28:39.622403 13337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/wal/instance:
uuid: "fed9c0217989495697d98fbb361fd67c"
format_stamp: "Formatted at 2026-06-20 14:28:39 on dist-test-slave-hfc3"
server_key: "84a785cf339a9d7380ea90996d4766f7"
server_key_iv: "1ad3a84caf1a980092752c3502430332"
server_key_version: "encryptionkey@0"
I20260620 14:28:39.623670 13337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:28:39.624329 13352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:39.624518 13337 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:28:39.624610 13337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/wal
uuid: "fed9c0217989495697d98fbb361fd67c"
format_stamp: "Formatted at 2026-06-20 14:28:39 on dist-test-slave-hfc3"
server_key: "84a785cf339a9d7380ea90996d4766f7"
server_key_iv: "1ad3a84caf1a980092752c3502430332"
server_key_version: "encryptionkey@0"
I20260620 14:28:39.624681 13337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:39.627373 13330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } attempt: 2
I20260620 14:28:39.627792 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.627644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39654)
0620 14:28:39.627711 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.627754 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:28:39.628227 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.628103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39664)
0620 14:28:39.628165 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.628200 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:28:39.628332 13330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36121: connect: Connection refused (error 111)
I20260620 14:28:39.644551 13337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:39.646963 13337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:39.647114 13337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:39.651247 13337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36121
I20260620 14:28:39.651321 13404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36121 every 8 connection(s)
I20260620 14:28:39.651674 13337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:28:39.652529 13405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:39.654127 13405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:39.654968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 13337
I20260620 14:28:39.655059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965454096380-392-0/minicluster-data/master-2/wal/instance
I20260620 14:28:39.655272 392 external_mini_cluster.cc:1428] Setting key ae8dafe519b0b759aac0bab3476d4cdd
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:28:39.664512 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.654921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.0.98.62:38591)
0620 14:28:39.654995 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.654998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.655241 (+ 243us) server_negotiation.cc:407] Connection header received
0620 14:28:39.655928 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.655938 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.655950 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.656008 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.656676 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.657263 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.658555 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.658766 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.658855 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.661337 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.661339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.661339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.661348 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.663029 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.663565 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.663566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.663566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.663616 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.663851 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.663852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.663852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.663959 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.664042 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.664341 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.664420 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:28:39.664832 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.654860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38591 (local address 127.0.0.1:49856)
0620 14:28:39.655111 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.655151 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:39.655817 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:39.656095 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:39.656099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:39.656428 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:39.656578 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.656583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.657320 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.657321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.658469 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.658472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.658881 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.658882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.659006 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.659579 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:39.659590 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:39.661245 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:39.663112 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.663117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.663119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.663484 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.663674 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.663675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.663676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.663795 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.664046 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:39.664053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:39.664195 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 14:28:39.664307 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:28:39.664469 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":151,"thread_start_us":77,"threads_started":1}
I20260620 14:28:39.665076 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.655702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.0.98.60:36121)
0620 14:28:39.656017 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.656022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.656038 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:28:39.656089 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.656091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.656109 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.656198 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.656886 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.657980 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.658826 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.659148 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.659242 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.661778 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.661785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.661786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.661793 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.663612 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.664116 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.664119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.664119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.664176 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.664387 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.664388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.664388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.664524 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.664699 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.664889 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.664968 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":96,"threads_started":1}
I20260620 14:28:39.665575 13405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:39.667752 13255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } attempt: 2
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:28:39.673072 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.665811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34191 (local address 127.0.0.1:48276)
0620 14:28:39.665999 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.666010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:39.666062 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:39.666196 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:39.666197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:39.666307 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:39.666405 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.666407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.667123 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.667124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.667658 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.667660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.668177 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.668178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.668255 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.668701 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:39.668706 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:39.669937 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:39.671958 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.671959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.671959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.672175 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.672434 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.672435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.672436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.672500 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.672777 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:39.672777 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:39.672819 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:28:39.672882 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:28:39.672977 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260620 14:28:39.673077 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.665889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.0.98.61:34191)
0620 14:28:39.665964 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.665966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.666050 (+ 84us) server_negotiation.cc:407] Connection header received
0620 14:28:39.666115 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.666116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.666127 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.666189 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.666486 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.667048 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.667741 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.668056 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.668123 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.670036 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.670038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.670038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.670047 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.671884 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.672288 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.672289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.672289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.672352 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.672579 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.672580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.672580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.672722 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.672783 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.672922 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.672994 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:28:39.673681 13405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:39.674171 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.665944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.0.98.62:38591)
0620 14:28:39.665997 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.665998 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.666008 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:28:39.666029 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.666030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.666039 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.666095 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.666711 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.667179 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.667921 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.668209 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.668268 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.670935 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.670937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.670937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.670945 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.672656 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.673103 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.673104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.673105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.673163 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.673416 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.673418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.673418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.673555 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.673724 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.674002 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.674075 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:28:39.675927 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.668009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39688)
0620 14:28:39.668052 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.668061 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:39.668091 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:39.668238 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:39.668239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:39.668394 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:39.668441 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.668443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.669235 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.669236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.670046 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.670049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.670595 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.670597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.670663 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.671066 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:39.671071 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:39.672627 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:39.674649 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.674650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.674650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.674889 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.675093 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.675094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.675094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.675167 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.675632 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:39.675633 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:39.675676 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:28:39.675745 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:28:39.675828 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:28:39.675979 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.668089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.0.98.60:36121)
0620 14:28:39.668123 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.668125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.668133 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:28:39.668153 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.668153 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.668163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.668216 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.668530 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.669129 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.670145 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.670504 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.670584 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.672703 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.672704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.672705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.672711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.674572 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.674965 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.674966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.674967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.675029 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.675399 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.675400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.675400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.675540 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.675633 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.675792 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.675869 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:28:39.677501 13255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0: Bootstrap starting.
I20260620 14:28:39.678224 13255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0: Neither blocks nor log segments found. Creating new log.
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:28:39.678591 13255 log.cc:824] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:39.679795 13255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0: No bootstrap required, opened a new log
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:28:39.681704 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.673969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39690)
0620 14:28:39.674009 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.674019 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:39.674056 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:39.674572 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:39.674573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:39.674703 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:39.674734 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.674735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.676185 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.676186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.676735 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.676737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.677053 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.677054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.677101 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.677510 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:39.677515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:39.678931 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:39.680685 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.680686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.680687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.680847 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.681059 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.681060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.681060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.681113 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.681399 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:39.681400 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:39.681438 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:28:39.681502 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:28:39.681601 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 14:28:39.682047 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.673981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39690 (local address 127.0.98.60:36121)
0620 14:28:39.674418 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.674419 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.674426 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:28:39.674443 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.674443 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.674472 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.674544 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.674818 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.676101 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.676801 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.676980 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.677054 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.679031 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.679032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.679033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.679039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.680588 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.680937 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.680938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.680939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.680995 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.681183 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.681184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.681185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.681316 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.681405 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.681532 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.681914 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":294,"server-negotiator.queue_time_us":418,"thread_start_us":88,"threads_started":1}
I20260620 14:28:39.682103 13255 raft_consensus.cc:348] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } }
I20260620 14:28:39.682390 13255 raft_consensus.cc:374] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:39.682507 13255 raft_consensus.cc:729] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71a395bb6a8432787a2ce53a3837ab0, State: Initialized, Role: FOLLOWER
I20260620 14:28:39.682708 13255 consensus_queue.cc:260] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } }
I20260620 14:28:39.683000 13412 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } } }
I20260620 14:28:39.683136 13412 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:39.683072 13255 sys_catalog.cc:564] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:39.683519 13405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c: Bootstrap starting.
I20260620 14:28:39.683898 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.675396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.98.61:34191)
0620 14:28:39.675431 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.675432 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.675440 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:28:39.675484 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.675484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.675493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.675549 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.676176 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.676785 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.677679 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.677992 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.678057 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.680816 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.680818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.680818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.680824 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.682553 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.682948 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.682949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.682949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.683010 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.683217 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.683218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.683219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.683341 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.683481 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.683704 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.683780 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:28:39.684514 13405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:39.684916 13405 log.cc:824] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c: Log is configured to *not* fsync() on all Append() calls
W20260620 14:28:39.685448 13423 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:28:39.685534 13423 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:28:39.686041 13405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c: No bootstrap required, opened a new log
I20260620 14:28:39.686250 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:28:39.687801 13405 raft_consensus.cc:348] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } }
I20260620 14:28:39.688006 13405 raft_consensus.cc:374] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:39.688071 13405 raft_consensus.cc:729] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed9c0217989495697d98fbb361fd67c, State: Initialized, Role: FOLLOWER
I20260620 14:28:39.688263 13405 consensus_queue.cc:260] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } }
I20260620 14:28:39.688653 13405 sys_catalog.cc:564] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:39.688117 13187 master_runner.cc:186] Error getting master registration for 127.0.98.61:34191: OK, instance_id { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" instance_seqno: 1781965719482982 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:28:39.690563 13187 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:28:39.690764 13443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:28:39.690871 13443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:28:39.692682 13337 master_runner.cc:186] Error getting master registration for 127.0.98.61:34191: OK, instance_id { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" instance_seqno: 1781965719482982 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:28:39.694998 13337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:28:39.695122 13432 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } } }
I20260620 14:28:39.695360 13432 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:39.699743 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.687984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.0.98.61:34191)
0620 14:28:39.688024 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.688026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.688185 (+ 159us) server_negotiation.cc:407] Connection header received
0620 14:28:39.688284 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.688285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.688296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.688325 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.689124 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.689921 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.691015 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.691303 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.691349 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.695362 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.695363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.695364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.695371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.697261 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.698067 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.698068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.698069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.698129 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.698327 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.698328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.698328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.698561 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.698692 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.699301 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.699398 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:28:39.701858 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.687971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49868 (local address 127.0.98.62:38591)
0620 14:28:39.688014 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.688016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.691787 (+ 3771us) server_negotiation.cc:407] Connection header received
0620 14:28:39.691882 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.691883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.691892 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.691922 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.693688 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.694412 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.695689 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.695976 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.696019 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.698671 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.698672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.698673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.698680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.700319 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.700780 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.700781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.700782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.700841 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.701225 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.701226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.701227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.701362 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.701455 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.701680 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.701762 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:28:39.707852 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.695231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.0.98.60:36121)
0620 14:28:39.695274 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.695276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.700488 (+ 5212us) server_negotiation.cc:407] Connection header received
0620 14:28:39.700585 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.700586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.700597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.700627 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.701277 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.702039 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.703057 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.703389 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.703430 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.704858 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.704859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.704860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.704867 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.706643 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.707023 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.707025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.707025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.707077 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.707265 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.707266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.707266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.707401 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.707473 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.707671 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.707751 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:28:39.719488 13330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } attempt: 3
Jun 20 14:28:39 dist-test-slave-hfc3 krb5kdc[13174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:28:39.728595 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.719778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36121 (local address 127.0.0.1:39718)
0620 14:28:39.719864 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:39.719877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:39.719914 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:39.720269 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:39.720270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:39.720475 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:39.720530 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.720532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.721481 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.721482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.722312 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:39.722314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.722839 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.722840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:39.722904 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.723442 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:39.723447 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:39.725032 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:39.727404 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.727406 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.727406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.727638 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.727856 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:39.727857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:39.727857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:39.727942 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:39.728248 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:39.728249 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:39.728300 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:28:39.728370 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:28:39.728488 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 14:28:39.728610 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:39.719986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.0.98.60:36121)
0620 14:28:39.720122 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:39.720124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:39.720132 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:28:39.720152 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:39.720153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:39.720163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:39.720240 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:39.720621 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.721385 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.722446 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:39.722752 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:39.722824 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:39.725129 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:39.725131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:39.725131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:39.725139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:39.727305 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.727715 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.727717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.727717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.727780 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:39.727994 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:39.727994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:39.727995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:39.728151 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:39.728215 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:39.728427 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 14:28:39.728514 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 14:28:39.730888 13330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3: Bootstrap starting.
I20260620 14:28:39.731683 13330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:39.732002 13330 log.cc:824] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:39.732935 13330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3: No bootstrap required, opened a new log
I20260620 14:28:39.735455 13330 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } }
I20260620 14:28:39.735615 13330 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:39.735651 13330 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5677806a07c4cb8a97d4bdb2dbfcef3, State: Initialized, Role: FOLLOWER
I20260620 14:28:39.735816 13330 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } }
I20260620 14:28:39.736176 13330 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:39.736133 13447 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } } }
I20260620 14:28:39.736333 13447 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:28:39.738051 13458 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:28:39.738160 13458 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:28:39.740067 13262 master_runner.cc:263] No leader master found from master a5677806a07c4cb8a97d4bdb2dbfcef3
I20260620 14:28:39.740120 13262 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:28:39.753786 13412 raft_consensus.cc:482] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:39.753903 13412 raft_consensus.cc:504] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } }
I20260620 14:28:39.754113 13412 leader_election.cc:288] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5677806a07c4cb8a97d4bdb2dbfcef3 (127.0.98.61:34191), fed9c0217989495697d98fbb361fd67c (127.0.98.60:36121)
I20260620 14:28:39.754783 13304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a71a395bb6a8432787a2ce53a3837ab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" is_pre_election: true
I20260620 14:28:39.754940 13304 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a71a395bb6a8432787a2ce53a3837ab0 in term 0.
I20260620 14:28:39.755347 13379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a71a395bb6a8432787a2ce53a3837ab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed9c0217989495697d98fbb361fd67c" is_pre_election: true
I20260620 14:28:39.755498 13379 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a71a395bb6a8432787a2ce53a3837ab0 in term 0.
I20260620 14:28:39.755736 13204 leader_election.cc:302] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5677806a07c4cb8a97d4bdb2dbfcef3, a71a395bb6a8432787a2ce53a3837ab0; no voters:
I20260620 14:28:39.756047 13412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:28:39.756108 13412 raft_consensus.cc:482] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:28:39.756129 13412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:39.757004 13412 raft_consensus.cc:504] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } }
I20260620 14:28:39.757134 13412 leader_election.cc:288] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [CANDIDATE]: Term 1 election: Requested vote from peers a5677806a07c4cb8a97d4bdb2dbfcef3 (127.0.98.61:34191), fed9c0217989495697d98fbb361fd67c (127.0.98.60:36121)
I20260620 14:28:39.757557 13379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a71a395bb6a8432787a2ce53a3837ab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed9c0217989495697d98fbb361fd67c"
I20260620 14:28:39.757634 13379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:39.758625 13379 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a71a395bb6a8432787a2ce53a3837ab0 in term 1.
I20260620 14:28:39.758891 13206 leader_election.cc:302] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71a395bb6a8432787a2ce53a3837ab0, fed9c0217989495697d98fbb361fd67c; no voters:
I20260620 14:28:39.759044 13304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a71a395bb6a8432787a2ce53a3837ab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3"
I20260620 14:28:39.759119 13304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:39.759974 13304 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a71a395bb6a8432787a2ce53a3837ab0 in term 1.
I20260620 14:28:39.760241 13412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:39.760416 13412 raft_consensus.cc:686] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [term 1 LEADER]: Becoming Leader. State: Replica: a71a395bb6a8432787a2ce53a3837ab0, State: Running, Role: LEADER
I20260620 14:28:39.760499 13412 consensus_queue.cc:237] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } }
I20260620 14:28:39.760855 13412 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a71a395bb6a8432787a2ce53a3837ab0. Latest consensus state: current_term: 1 leader_uuid: "a71a395bb6a8432787a2ce53a3837ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } } }
I20260620 14:28:39.760918 13412 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:39.761224 13462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:39.761454 13462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:39.763442 13379 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [term 1 FOLLOWER]: Refusing update from remote peer a71a395bb6a8432787a2ce53a3837ab0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:39.764720 13412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:39.766208 13432 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a71a395bb6a8432787a2ce53a3837ab0. Latest consensus state: current_term: 1 leader_uuid: "a71a395bb6a8432787a2ce53a3837ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } } }
I20260620 14:28:39.766297 13432 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:39.766764 13304 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [term 1 FOLLOWER]: Refusing update from remote peer a71a395bb6a8432787a2ce53a3837ab0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:39.767088 13412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:39.768414 13447 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a71a395bb6a8432787a2ce53a3837ab0. Latest consensus state: current_term: 1 leader_uuid: "a71a395bb6a8432787a2ce53a3837ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } } }
I20260620 14:28:39.768514 13447 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:39.769907 13412 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a71a395bb6a8432787a2ce53a3837ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } } }
I20260620 14:28:39.769994 13412 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:39.770567 13447 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a71a395bb6a8432787a2ce53a3837ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } } }
I20260620 14:28:39.770651 13447 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:39.770907 13463 mvcc.cc:205] Tried to move back new op lower bound from 7298931588148494336 to 7298931588139331584. Current Snapshot: MvccSnapshot[applied={T|T < 7298931588148494336}]
I20260620 14:28:39.771124 13466 mvcc.cc:205] Tried to move back new op lower bound from 7298931588148494336 to 7298931588139331584. Current Snapshot: MvccSnapshot[applied={T|T < 7298931588148494336}]
I20260620 14:28:39.772188 13412 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a71a395bb6a8432787a2ce53a3837ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } } }
I20260620 14:28:39.772272 13412 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:39.772846 13432 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a71a395bb6a8432787a2ce53a3837ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71a395bb6a8432787a2ce53a3837ab0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38591 } } peers { permanent_uuid: "a5677806a07c4cb8a97d4bdb2dbfcef3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34191 } } peers { permanent_uuid: "fed9c0217989495697d98fbb361fd67c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36121 } } }
I20260620 14:28:39.772943 13432 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:39.773303 13464 mvcc.cc:205] Tried to move back new op lower bound from 7298931588148494336 to 7298931588139331584. Current Snapshot: MvccSnapshot[applied={T|T < 7298931588148494336}]
I20260620 14:28:39.773941 13462 catalog_manager.cc:1347] Generated new cluster ID: f6584f438b764c4097a13bddd2fe8db3
I20260620 14:28:39.773988 13462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:39.789021 13462 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:28:39.789875 13462 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:39.799300 13462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a71a395bb6a8432787a2ce53a3837ab0: Generated new TSK 0
I20260620 14:28:39.799630 13462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:28:40.691354 13443 catalog_manager.cc:1259] Loaded cluster ID: f6584f438b764c4097a13bddd2fe8db3
I20260620 14:28:40.691434 13443 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c: loading cluster ID for follower catalog manager: success
I20260620 14:28:40.692240 13443 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c: acquiring CA information for follower catalog manager: success
I20260620 14:28:40.692510 13443 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fed9c0217989495697d98fbb361fd67c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:28:40.738652 13458 catalog_manager.cc:1259] Loaded cluster ID: f6584f438b764c4097a13bddd2fe8db3
I20260620 14:28:40.738729 13458 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3: loading cluster ID for follower catalog manager: success
I20260620 14:28:40.739567 13458 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3: acquiring CA information for follower catalog manager: success
I20260620 14:28:40.739876 13458 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5677806a07c4cb8a97d4bdb2dbfcef3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:28:40.858784 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 13187
I20260620 14:28:40.864351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 13262
I20260620 14:28:40.869513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 13337
2026-06-20T14:28:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2839 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:28:40 dist-test-slave-hfc3 krb5kdc[13481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:28:40 dist-test-slave-hfc3 krb5kdc[13481](info): set up 2 sockets
Jun 20 14:28:40 dist-test-slave-hfc3 krb5kdc[13481](info): commencing operation
krb5kdc: starting...
W20260620 14:28:41.922426 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:28:42 dist-test-slave-hfc3 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:28:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:28:42.045527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42427
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:42.129570 13494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:42.129848 13494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:42.129889 13494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:42.131210 13494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:42.131296 13494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:42.131321 13494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:42.131350 13494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:42.131383 13494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:42.132913 13494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:42.133211 13494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:42.133472 13494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:42.138661 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:42.138657 13502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:42.138657 13499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:42.138964 13494 server_base.cc:1034] running on GCE node
I20260620 14:28:42.139245 13494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:42.139604 13494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:42.140758 13494 hybrid_clock.cc:648] HybridClock initialized: now 1781965722140736 us; error 45 us; skew 500 ppm
Jun 20 14:28:42 dist-test-slave-hfc3 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:42.148316 13494 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:42.148847 13494 webserver.cc:460] Webserver started at http://127.0.98.62:37671/ using document root <none> and password file <none>
I20260620 14:28:42.149123 13494 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:42.149204 13494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:42.149362 13494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:42.150600 13494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "a3e02347ce16453db64bc13b9df8350d"
format_stamp: "Formatted at 2026-06-20 14:28:42 on dist-test-slave-hfc3"
server_key: "91b4404b86ce20a8dfda9d3582dfaa7a"
server_key_iv: "ee4cc55704a1f0088fb26b5c8fbb50ce"
server_key_version: "encryptionkey@0"
I20260620 14:28:42.150980 13494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "a3e02347ce16453db64bc13b9df8350d"
format_stamp: "Formatted at 2026-06-20 14:28:42 on dist-test-slave-hfc3"
server_key: "91b4404b86ce20a8dfda9d3582dfaa7a"
server_key_iv: "ee4cc55704a1f0088fb26b5c8fbb50ce"
server_key_version: "encryptionkey@0"
I20260620 14:28:42.152393 13494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:28:42.153209 13509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:42.153403 13494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:42.153537 13494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "a3e02347ce16453db64bc13b9df8350d"
format_stamp: "Formatted at 2026-06-20 14:28:42 on dist-test-slave-hfc3"
server_key: "91b4404b86ce20a8dfda9d3582dfaa7a"
server_key_iv: "ee4cc55704a1f0088fb26b5c8fbb50ce"
server_key_version: "encryptionkey@0"
I20260620 14:28:42.153620 13494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:42.168905 13494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:42.171454 13494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:42.171638 13494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:42.175806 13494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42427
I20260620 14:28:42.175853 13561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42427 every 8 connection(s)
I20260620 14:28:42.176245 13494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:42.177170 13562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:42.179569 13562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d: Bootstrap starting.
I20260620 14:28:42.180119 13562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:42.180395 13562 log.cc:824] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:42.181221 13562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d: No bootstrap required, opened a new log
I20260620 14:28:42.181288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 13494
I20260620 14:28:42.181457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:28:42.181670 392 external_mini_cluster.cc:1428] Setting key bb9e6a61ace40a82f5f0b71fa8f58050
I20260620 14:28:42.183326 13562 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e02347ce16453db64bc13b9df8350d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42427 } }
I20260620 14:28:42.183509 13562 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:42.183553 13562 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e02347ce16453db64bc13b9df8350d, State: Initialized, Role: FOLLOWER
I20260620 14:28:42.183735 13562 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e02347ce16453db64bc13b9df8350d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42427 } }
I20260620 14:28:42.183838 13562 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:42.183872 13562 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:42.183926 13562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:42.184995 13562 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e02347ce16453db64bc13b9df8350d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42427 } }
I20260620 14:28:42.185088 13562 leader_election.cc:302] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3e02347ce16453db64bc13b9df8350d; no voters:
I20260620 14:28:42.185262 13562 leader_election.cc:288] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:28:42.185312 13567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:42.185496 13567 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [term 1 LEADER]: Becoming Leader. State: Replica: a3e02347ce16453db64bc13b9df8350d, State: Running, Role: LEADER
I20260620 14:28:42.185551 13562 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:42.185622 13567 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e02347ce16453db64bc13b9df8350d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42427 } }
I20260620 14:28:42.186093 13568 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3e02347ce16453db64bc13b9df8350d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e02347ce16453db64bc13b9df8350d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42427 } } }
I20260620 14:28:42.186169 13568 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:42.186136 13569 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3e02347ce16453db64bc13b9df8350d. Latest consensus state: current_term: 1 leader_uuid: "a3e02347ce16453db64bc13b9df8350d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e02347ce16453db64bc13b9df8350d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42427 } } }
I20260620 14:28:42.186241 13569 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d [sys.catalog]: This master's current role is: LEADER
W20260620 14:28:42.187583 13583 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:28:42.187647 13583 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:28:42.187691 13575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:42.187829 13575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:42.189538 13575 catalog_manager.cc:1347] Generated new cluster ID: 8b1d10870c4a484ab3666a98566229d2
I20260620 14:28:42.189603 13575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:28:42 dist-test-slave-hfc3 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:42.193763 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:42.182195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56386 (local address 127.0.98.62:42427)
0620 14:28:42.182396 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:42.182405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:42.182440 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:28:42.183305 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:42.183312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:42.183361 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:42.183478 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:42.184347 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:42.185455 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:42.186632 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:42.187341 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:42.187452 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:42.190267 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:42.190270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:42.190271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:42.190283 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:42.192150 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:42.192537 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:42.192540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:42.192540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:42.192656 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:42.192884 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:42.192885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:42.192885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:42.193140 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:42.193333 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:42.193403 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:28:42.193558 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20260620 14:28:42.194695 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:28:42.203233 13575 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:28:42.203760 13575 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:42.208833 13575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3e02347ce16453db64bc13b9df8350d: Generated new TSK 0
I20260620 14:28:42.209048 13575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:28:42.248656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 13494
2026-06-20T14:28:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1377 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:28:42 dist-test-slave-hfc3 krb5kdc[13589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:28:42 dist-test-slave-hfc3 krb5kdc[13589](info): set up 2 sockets
Jun 20 14:28:42 dist-test-slave-hfc3 krb5kdc[13589](info): commencing operation
krb5kdc: starting...
W20260620 14:28:43.543429 392 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:28:43 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:28:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:28:43.697216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46071
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:43.780494 13602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:43.780717 13602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:43.780741 13602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:43.781836 13602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:43.781872 13602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:43.781888 13602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:43.781899 13602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:43.781912 13602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:43.783350 13602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:43.783548 13602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:43.783726 13602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:43.788736 13607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:43.788805 13610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:43.788736 13608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:43.789002 13602 server_base.cc:1034] running on GCE node
I20260620 14:28:43.789270 13602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:43.789579 13602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:43.790750 13602 hybrid_clock.cc:648] HybridClock initialized: now 1781965723790741 us; error 36 us; skew 500 ppm
Jun 20 14:28:43 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:43.793341 13602 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:43.793819 13602 webserver.cc:460] Webserver started at http://127.0.98.62:36441/ using document root <none> and password file <none>
I20260620 14:28:43.794049 13602 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:43.794126 13602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:43.794286 13602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:43.795387 13602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc"
format_stamp: "Formatted at 2026-06-20 14:28:43 on dist-test-slave-hfc3"
server_key: "53edfccede37b18677e836ea12bc4376"
server_key_iv: "d5dff4c3a8fbd1a9af84a714c74f0cfd"
server_key_version: "encryptionkey@0"
I20260620 14:28:43.795774 13602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc"
format_stamp: "Formatted at 2026-06-20 14:28:43 on dist-test-slave-hfc3"
server_key: "53edfccede37b18677e836ea12bc4376"
server_key_iv: "d5dff4c3a8fbd1a9af84a714c74f0cfd"
server_key_version: "encryptionkey@0"
I20260620 14:28:43.797300 13602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:28:43.798192 13617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:43.798429 13602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:28:43.798548 13602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc"
format_stamp: "Formatted at 2026-06-20 14:28:43 on dist-test-slave-hfc3"
server_key: "53edfccede37b18677e836ea12bc4376"
server_key_iv: "d5dff4c3a8fbd1a9af84a714c74f0cfd"
server_key_version: "encryptionkey@0"
I20260620 14:28:43.798650 13602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:43.823058 13602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:43.825557 13602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:43.825738 13602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:43.829967 13602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46071
I20260620 14:28:43.830016 13669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46071 every 8 connection(s)
I20260620 14:28:43.830466 13602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:43.831498 13670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:43.832593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 13602
I20260620 14:28:43.832710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:28:43.832890 392 external_mini_cluster.cc:1428] Setting key 79c7d6e4f41d9bac5dc21cc03896695c
I20260620 14:28:43.834209 13670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Bootstrap starting.
I20260620 14:28:43.835037 13670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:43.835341 13670 log.cc:824] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:43.836026 13670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: No bootstrap required, opened a new log
I20260620 14:28:43.838459 13670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260620 14:28:43.838663 13670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:43.838707 13670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbe4b02cb244bf5a4b6806a7d3d95fc, State: Initialized, Role: FOLLOWER
I20260620 14:28:43.838855 13670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260620 14:28:43.838944 13670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:43.838981 13670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:43.839035 13670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:43.839843 13670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260620 14:28:43.839969 13670 leader_election.cc:302] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dbe4b02cb244bf5a4b6806a7d3d95fc; no voters:
I20260620 14:28:43.840138 13670 leader_election.cc:288] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:28:43.840178 13675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:43.840297 13675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 1 LEADER]: Becoming Leader. State: Replica: 1dbe4b02cb244bf5a4b6806a7d3d95fc, State: Running, Role: LEADER
Jun 20 14:28:43 dist-test-slave-hfc3 krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:43.840443 13670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:43.840983 13675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260620 14:28:43.841396 13675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dbe4b02cb244bf5a4b6806a7d3d95fc. Latest consensus state: current_term: 1 leader_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } }
I20260620 14:28:43.841846 13675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: This master's current role is: LEADER
W20260620 14:28:43.842561 13689 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:28:43.842680 13689 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:28:43.842831 13692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:43.841760 13676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } }
I20260620 14:28:43.842994 13676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:43.843055 13692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:43.844594 13692 catalog_manager.cc:1347] Generated new cluster ID: 894566a759cf4fd9859074cbbf675e43
I20260620 14:28:43.844655 13692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:43.845986 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:43.833463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.0.98.62:46071)
0620 14:28:43.833648 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:43.833655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:43.833680 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:28:43.834117 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:43.834123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:43.834175 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:43.834291 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:43.835183 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:43.836214 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:43.837239 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:43.837784 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:43.837870 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:43.841039 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:43.841043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:43.841044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:43.841059 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:43.843882 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:43.844289 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:43.844293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:43.844293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:43.844428 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:43.844689 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:43.844690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:43.844690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:43.845008 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:43.845152 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:43.845615 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 14:28:43.845751 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260620 14:28:43.860915 13692 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:28:43.862166 13692 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:43.867041 13692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Generated new TSK 0
I20260620 14:28:43.867259 13692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:28:43.897120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46071
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:43.982450 13696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:43.982667 13696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:43.982692 13696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:43.983877 13696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:43.983922 13696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:43.983949 13696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:28:43.985463 13696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:43.985711 13696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:43.985965 13696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:43.991125 13704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:43.991338 13696 server_base.cc:1034] running on GCE node
W20260620 14:28:43.991143 13702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:43.991147 13701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:43.991641 13696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:43.991868 13696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:43.993074 13696 hybrid_clock.cc:648] HybridClock initialized: now 1781965723993046 us; error 44 us; skew 500 ppm
Jun 20 14:28:43 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:43.995818 13696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:28:43.996289 13696 webserver.cc:460] Webserver started at http://127.0.98.1:40879/ using document root <none> and password file <none>
I20260620 14:28:43.996536 13696 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:43.996618 13696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:43.996775 13696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:43.997913 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "59cc386a6f484094b8b40f15b9977a65"
format_stamp: "Formatted at 2026-06-20 14:28:43 on dist-test-slave-hfc3"
server_key: "dec28a6ac82a0219ae4ef4770907db6d"
server_key_iv: "95bf7977b675e7c418acd17574e84e62"
server_key_version: "encryptionkey@0"
I20260620 14:28:43.998302 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "59cc386a6f484094b8b40f15b9977a65"
format_stamp: "Formatted at 2026-06-20 14:28:43 on dist-test-slave-hfc3"
server_key: "dec28a6ac82a0219ae4ef4770907db6d"
server_key_iv: "95bf7977b675e7c418acd17574e84e62"
server_key_version: "encryptionkey@0"
I20260620 14:28:43.999724 13696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:44.000515 13711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:44.000701 13696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:28:44.000814 13696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "59cc386a6f484094b8b40f15b9977a65"
format_stamp: "Formatted at 2026-06-20 14:28:43 on dist-test-slave-hfc3"
server_key: "dec28a6ac82a0219ae4ef4770907db6d"
server_key_iv: "95bf7977b675e7c418acd17574e84e62"
server_key_version: "encryptionkey@0"
I20260620 14:28:44.000897 13696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:44.032045 13696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:44.034176 13696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:44.034442 13696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:44.034722 13696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:44.035061 13696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:44.035122 13696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:44.035166 13696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:44.035212 13696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:44.041600 13696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44759
I20260620 14:28:44.041689 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44759 every 8 connection(s)
I20260620 14:28:44.042088 13696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:28:44.042876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 13696
I20260620 14:28:44.043000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:28:44.043212 392 external_mini_cluster.cc:1428] Setting key f4e8a040e20028338464de5d232df147
Jun 20 14:28:44 dist-test-slave-hfc3 krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:28:44.053459 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.042980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35325 (local address 127.0.98.62:46071)
0620 14:28:44.043038 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:44.043040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:44.043317 (+ 277us) server_negotiation.cc:407] Connection header received
0620 14:28:44.043969 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:44.043971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:44.043982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:44.044050 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:44.045410 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.045956 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.047402 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.047608 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.047704 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.050255 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:44.050257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:44.050257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:44.050266 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:44.051891 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:44.052355 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:44.052357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:44.052358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:44.052424 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:44.052704 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:44.052705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:44.052705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:44.052846 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:44.052929 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:44.053242 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 14:28:44.053360 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:28:44.053653 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.042901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:35325)
0620 14:28:44.043142 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:44.043196 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:44.043859 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:44.044133 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:44.044137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:44.045014 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:44.045307 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.045314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.046057 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.046058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.047315 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.047318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.047717 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.047719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.047856 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.048454 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:44.048464 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:44.050146 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:44.052004 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:44.052013 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:44.052015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:44.052257 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:44.052495 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:44.052496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:44.052496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:44.052623 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:44.052931 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:44.052939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:44.053095 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 14:28:44.053212 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:28:44.053387 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260620 14:28:44.054260 13825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46071
I20260620 14:28:44.054401 13825 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:44.054615 13825 heartbeater.cc:511] Master 127.0.98.62:46071 requested a full tablet report, sending...
I20260620 14:28:44.055188 13634 ts_manager.cc:194] Registered new tserver with Master: 59cc386a6f484094b8b40f15b9977a65 (127.0.98.1:44759)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:28:44.056144 13634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:28:44.073699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46071
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:44.159219 13830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:44.159473 13830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:44.159525 13830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:44.160749 13830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:44.160830 13830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:44.160876 13830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:28:44.162401 13830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:44.162703 13830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:44.162962 13830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:44.168215 13838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:44.168278 13836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:44.168290 13835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:44.168543 13830 server_base.cc:1034] running on GCE node
I20260620 14:28:44.168777 13830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:44.169140 13830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:44.170324 13830 hybrid_clock.cc:648] HybridClock initialized: now 1781965724170313 us; error 49 us; skew 500 ppm
Jun 20 14:28:44 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:44.173038 13830 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:28:44.173521 13830 webserver.cc:460] Webserver started at http://127.0.98.2:39701/ using document root <none> and password file <none>
I20260620 14:28:44.173784 13830 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:44.173859 13830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:44.174019 13830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:44.175180 13830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "6115885c5692438c88137fac7d081af1"
format_stamp: "Formatted at 2026-06-20 14:28:44 on dist-test-slave-hfc3"
server_key: "25d4c9c217632634df02f9d600fea3dc"
server_key_iv: "4056189b8d6b9d2973009681ff8edf94"
server_key_version: "encryptionkey@0"
I20260620 14:28:44.175554 13830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6115885c5692438c88137fac7d081af1"
format_stamp: "Formatted at 2026-06-20 14:28:44 on dist-test-slave-hfc3"
server_key: "25d4c9c217632634df02f9d600fea3dc"
server_key_iv: "4056189b8d6b9d2973009681ff8edf94"
server_key_version: "encryptionkey@0"
I20260620 14:28:44.176925 13830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:28:44.177724 13845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:44.177989 13830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:28:44.178107 13830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "6115885c5692438c88137fac7d081af1"
format_stamp: "Formatted at 2026-06-20 14:28:44 on dist-test-slave-hfc3"
server_key: "25d4c9c217632634df02f9d600fea3dc"
server_key_iv: "4056189b8d6b9d2973009681ff8edf94"
server_key_version: "encryptionkey@0"
I20260620 14:28:44.178200 13830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:44.196902 13830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:44.199416 13830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:44.199581 13830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:44.199820 13830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:44.200160 13830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:44.200219 13830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:44.200263 13830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:44.200289 13830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:44.206809 13830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36991
I20260620 14:28:44.206883 13958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36991 every 8 connection(s)
I20260620 14:28:44.207245 13830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:28:44.209659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 13830
I20260620 14:28:44.209754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:28:44.209950 392 external_mini_cluster.cc:1428] Setting key 0ffee3e83d490c1ef528d3fc2ad489f6
Jun 20 14:28:44 dist-test-slave-hfc3 krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:44.218259 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.208067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58089 (local address 127.0.98.62:46071)
0620 14:28:44.208160 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:44.208162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:44.208416 (+ 254us) server_negotiation.cc:407] Connection header received
0620 14:28:44.208936 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:44.208937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:44.208947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:44.209028 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:44.210304 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.210965 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.212013 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.212328 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.212388 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.215098 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:44.215100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:44.215100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:44.215107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:44.216645 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:44.217162 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:44.217163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:44.217163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:44.217215 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:44.217518 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:44.217519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:44.217519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:44.217659 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:44.217712 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:44.218109 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 14:28:44.218192 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:28:44.218564 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.208045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:58089)
0620 14:28:44.208253 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:44.208297 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:44.208839 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:44.209097 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:44.209100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:44.209895 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:44.210191 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.210204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.211063 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.211064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.211918 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.211921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.212423 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.212425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.212630 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.213189 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:44.213199 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:44.214989 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:44.216746 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:44.216755 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:44.216757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:44.217075 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:44.217287 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:44.217288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:44.217288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:44.217446 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:44.217735 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:44.217743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:44.217925 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 14:28:44.218063 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 14:28:44.218259 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":135,"threads_started":1}
I20260620 14:28:44.219166 13959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46071
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:28:44.219288 13959 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:44.219503 13959 heartbeater.cc:511] Master 127.0.98.62:46071 requested a full tablet report, sending...
I20260620 14:28:44.219903 13634 ts_manager.cc:194] Registered new tserver with Master: 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991)
I20260620 14:28:44.220337 13634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:28:44.237825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46071
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:44.320664 13964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:44.320895 13964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:44.320920 13964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:44.322036 13964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:44.322074 13964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:44.322100 13964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:28:44.323625 13964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:44.323832 13964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:44.324009 13964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:44.329391 13970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:44.329385 13972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:44.329417 13964 server_base.cc:1034] running on GCE node
W20260620 14:28:44.329385 13969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:44.329859 13964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:44.330067 13964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:44.331215 13964 hybrid_clock.cc:648] HybridClock initialized: now 1781965724331206 us; error 31 us; skew 500 ppm
Jun 20 14:28:44 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:44.333730 13964 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:28:44.334120 13964 webserver.cc:460] Webserver started at http://127.0.98.3:37827/ using document root <none> and password file <none>
I20260620 14:28:44.334424 13964 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:44.334476 13964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:44.334623 13964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:44.335705 13964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "bda0b3ebe98d4d78915c4eb59126fcd9"
format_stamp: "Formatted at 2026-06-20 14:28:44 on dist-test-slave-hfc3"
server_key: "f568411950b4aa164147656f7b7b2353"
server_key_iv: "822509907c513478a81c5d2fe9803187"
server_key_version: "encryptionkey@0"
I20260620 14:28:44.336112 13964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bda0b3ebe98d4d78915c4eb59126fcd9"
format_stamp: "Formatted at 2026-06-20 14:28:44 on dist-test-slave-hfc3"
server_key: "f568411950b4aa164147656f7b7b2353"
server_key_iv: "822509907c513478a81c5d2fe9803187"
server_key_version: "encryptionkey@0"
I20260620 14:28:44.337476 13964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:28:44.338284 13979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:44.338510 13964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:28:44.338658 13964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "bda0b3ebe98d4d78915c4eb59126fcd9"
format_stamp: "Formatted at 2026-06-20 14:28:44 on dist-test-slave-hfc3"
server_key: "f568411950b4aa164147656f7b7b2353"
server_key_iv: "822509907c513478a81c5d2fe9803187"
server_key_version: "encryptionkey@0"
I20260620 14:28:44.338738 13964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:44.355839 13964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:44.357926 13964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:44.358109 13964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:44.358428 13964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:44.358776 13964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:44.358835 13964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:44.358879 13964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:44.358906 13964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:44.364734 13964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40199
I20260620 14:28:44.364830 14092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40199 every 8 connection(s)
I20260620 14:28:44.365163 13964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:28:44 dist-test-slave-hfc3 krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:44.373610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 13964
I20260620 14:28:44.373713 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:28:44.373910 392 external_mini_cluster.cc:1428] Setting key df426b337a9e803c6b6d4f4551510979
I20260620 14:28:44.375053 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.365944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49139 (local address 127.0.98.62:46071)
0620 14:28:44.365992 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:44.365993 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:44.366188 (+ 195us) server_negotiation.cc:407] Connection header received
0620 14:28:44.366838 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:44.366839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:44.366849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:44.366907 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:44.368042 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.368534 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.369577 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.369769 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.369833 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.371993 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:44.371994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:44.371994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:44.372001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:44.373295 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:44.373851 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:44.373852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:44.373852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:44.373917 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:44.374243 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:44.374244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:44.374245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:44.374428 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:44.374498 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:44.374860 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 14:28:44.374944 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:28:44.375278 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.365894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:49139)
0620 14:28:44.366063 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:44.366108 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:44.366690 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:44.366998 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:44.367002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:44.367738 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:44.367953 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.367957 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.368631 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.368632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.369440 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.369442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.369854 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.369854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.369955 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.370544 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:44.370553 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:44.371909 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:44.373372 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:44.373378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:44.373380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:44.373719 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:44.374008 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:44.374009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:44.374009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:44.374179 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:44.374507 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:44.374514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:44.374696 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 14:28:44.374816 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:28:44.374991 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260620 14:28:44.375774 14093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46071
I20260620 14:28:44.375859 14093 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:44.376060 14093 heartbeater.cc:511] Master 127.0.98.62:46071 requested a full tablet report, sending...
I20260620 14:28:44.376413 13634 ts_manager.cc:194] Registered new tserver with Master: bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199)
I20260620 14:28:44.376808 13634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49139
I20260620 14:28:44.385809 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:28:44.393812 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.387260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38264 (local address 127.0.98.62:46071)
0620 14:28:44.387335 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:44.387337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:44.387421 (+ 84us) server_negotiation.cc:407] Connection header received
0620 14:28:44.387454 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:44.387455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:44.387465 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:44.387531 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:44.388136 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.388634 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.389342 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.389524 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.389634 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.391026 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:44.391028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:44.391028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:44.391036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:44.392624 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:44.392964 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:44.392965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:44.392965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:44.393025 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:44.393203 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:44.393203 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:44.393204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:44.393362 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:44.393449 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:44.393535 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 14:28:44.393645 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:28:44.395744 13634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:28:44.397485 13634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:28:44.404896 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.400014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40199 (local address 127.0.0.1:45378)
0620 14:28:44.400553 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:44.400565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:44.400638 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:44.400868 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:44.400869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:44.400870 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:44.400910 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.400912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.402417 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.402418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.403583 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.403587 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.404543 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.404544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.404635 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.404657 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:28:44.404733 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:28:44.404793 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":46,"thread_start_us":154,"threads_started":1}
I20260620 14:28:44.404978 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.400333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.0.98.1:44759)
0620 14:28:44.400561 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:44.400565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:44.400575 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:28:44.400643 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:44.400645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:44.400662 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:44.400961 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:44.401062 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.402229 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.403929 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.404620 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.404683 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.404746 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:44.404839 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:28:44.404877 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":86,"threads_started":1}
I20260620 14:28:44.405035 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.399979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44759 (local address 127.0.0.1:33872)
0620 14:28:44.400161 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:44.400205 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:44.400275 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:44.400793 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:44.400797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:44.400800 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:44.400878 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.400880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.402380 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.402381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.403832 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.403835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.404697 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.404698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.404766 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.404770 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:28:44.404825 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:28:44.404872 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":105,"thread_start_us":59,"threads_started":1}
I20260620 14:28:44.405284 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.400014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36991 (local address 127.0.0.1:48884)
0620 14:28:44.400193 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:44.400206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:44.400274 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:44.400999 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:44.401000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:44.401003 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:44.401056 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.401058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.402229 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.402230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.403839 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.403841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.404701 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.404702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.404775 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.404779 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:28:44.404810 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 14:28:44.404846 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":57,"thread_start_us":119,"threads_started":1}
I20260620 14:28:44.405414 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.400567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.0.98.2:36991)
0620 14:28:44.400793 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:44.400798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:44.400818 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:28:44.400867 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:44.400870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:44.400886 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:44.401154 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:44.401257 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.402100 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.403948 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.404593 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.404933 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.405008 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:44.405037 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 14:28:44.405092 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
I20260620 14:28:44.405939 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.400316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45378 (local address 127.0.98.3:40199)
0620 14:28:44.400542 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:44.400548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:44.400642 (+ 94us) server_negotiation.cc:407] Connection header received
0620 14:28:44.400723 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:44.400725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:44.400743 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:44.400984 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:44.401061 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.402258 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.403689 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.404447 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.404522 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.404582 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:44.405806 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0620 14:28:44.405851 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
I20260620 14:28:44.405865 13759 tablet_service.cc:1467] Processing CreateTablet for tablet 3fbc3493f8424fdaad7ca13f4afedd23 (DEFAULT_TABLE table=test-table [id=e4dd8887a2cb49fc810bbe8596ad2733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:44.406232 13759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fbc3493f8424fdaad7ca13f4afedd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:44.406215 13893 tablet_service.cc:1467] Processing CreateTablet for tablet 3fbc3493f8424fdaad7ca13f4afedd23 (DEFAULT_TABLE table=test-table [id=e4dd8887a2cb49fc810bbe8596ad2733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:44.406636 13893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fbc3493f8424fdaad7ca13f4afedd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:44.406728 14027 tablet_service.cc:1467] Processing CreateTablet for tablet 3fbc3493f8424fdaad7ca13f4afedd23 (DEFAULT_TABLE table=test-table [id=e4dd8887a2cb49fc810bbe8596ad2733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:44.407052 14027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fbc3493f8424fdaad7ca13f4afedd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:44.409528 14110 tablet_bootstrap.cc:492] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1: Bootstrap starting.
I20260620 14:28:44.410132 14111 tablet_bootstrap.cc:492] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Bootstrap starting.
I20260620 14:28:44.410398 14112 tablet_bootstrap.cc:492] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9: Bootstrap starting.
I20260620 14:28:44.410607 14110 tablet_bootstrap.cc:654] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:44.411031 14111 tablet_bootstrap.cc:654] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:44.411123 14112 tablet_bootstrap.cc:654] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:44.411307 14110 log.cc:824] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:44.411715 14111 log.cc:824] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:44.411763 14112 log.cc:824] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:44.412356 14110 tablet_bootstrap.cc:492] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1: No bootstrap required, opened a new log
I20260620 14:28:44.412467 14110 ts_tablet_manager.cc:1397] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:28:44.412510 14111 tablet_bootstrap.cc:492] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: No bootstrap required, opened a new log
I20260620 14:28:44.412511 14112 tablet_bootstrap.cc:492] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9: No bootstrap required, opened a new log
I20260620 14:28:44.412570 14112 ts_tablet_manager.cc:1397] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:44.412570 14111 ts_tablet_manager.cc:1397] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:28:44.414115 14110 raft_consensus.cc:348] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } } peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260620 14:28:44.414305 14110 raft_consensus.cc:374] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:44.414381 14110 raft_consensus.cc:729] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6115885c5692438c88137fac7d081af1, State: Initialized, Role: FOLLOWER
I20260620 14:28:44.414549 14110 consensus_queue.cc:260] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } } peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260620 14:28:44.414634 14112 raft_consensus.cc:348] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } } peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260620 14:28:44.414634 14111 raft_consensus.cc:348] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } } peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260620 14:28:44.414813 14111 raft_consensus.cc:374] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:44.414813 14112 raft_consensus.cc:374] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:44.414858 14111 raft_consensus.cc:729] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cc386a6f484094b8b40f15b9977a65, State: Initialized, Role: FOLLOWER
I20260620 14:28:44.414858 14112 raft_consensus.cc:729] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda0b3ebe98d4d78915c4eb59126fcd9, State: Initialized, Role: FOLLOWER
I20260620 14:28:44.414861 14110 ts_tablet_manager.cc:1428] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:28:44.414988 13959 heartbeater.cc:503] Master 127.0.98.62:46071 was elected leader, sending a full tablet report...
I20260620 14:28:44.414970 14111 consensus_queue.cc:260] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } } peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260620 14:28:44.414990 14112 consensus_queue.cc:260] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } } peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260620 14:28:44.415237 14112 ts_tablet_manager.cc:1428] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:28:44.415251 14111 ts_tablet_manager.cc:1428] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:28:44.415362 13825 heartbeater.cc:503] Master 127.0.98.62:46071 was elected leader, sending a full tablet report...
I20260620 14:28:44.415635 14093 heartbeater.cc:503] Master 127.0.98.62:46071 was elected leader, sending a full tablet report...
W20260620 14:28:44.457924 13960 tablet.cc:2369] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:28:44.543102 13826 tablet.cc:2369] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:28:44.615805 14094 tablet.cc:2369] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:28:44.784672 14116 raft_consensus.cc:482] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:44.784791 14116 raft_consensus.cc:504] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } } peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260620 14:28:44.785239 14116 leader_election.cc:288] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59cc386a6f484094b8b40f15b9977a65 (127.0.98.1:44759), bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199)
I20260620 14:28:44.788447 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.785532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55371 (local address 127.0.98.1:44759)
0620 14:28:44.785620 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:44.785623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:44.785636 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:28:44.785697 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:44.785698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:44.785734 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:44.785771 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:44.786044 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.786655 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.787683 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.788043 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.788168 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.788179 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:44.788336 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:28:44.788374 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:28:44.788465 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.785414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44759 (local address 127.0.98.2:55371)
0620 14:28:44.785552 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:44.785563 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:44.785626 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:44.785851 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:44.785852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:44.785854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:44.785943 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.785945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.786772 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.786773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.787586 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.787588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.788170 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.788171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.788245 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.788249 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:28:44.788327 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:28:44.788376 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":22,"thread_start_us":58,"threads_started":1}
I20260620 14:28:44.788632 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.785417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40199 (local address 127.0.98.2:59005)
0620 14:28:44.785599 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:44.785608 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:44.785653 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:44.786046 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:44.786048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:44.786050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:44.786113 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.786115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.786902 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.786903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.787706 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:44.787708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.788365 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.788366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:44.788437 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.788440 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:28:44.788504 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:28:44.788547 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":60,"threads_started":1}
I20260620 14:28:44.788617 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.785701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59005 (local address 127.0.98.3:40199)
0620 14:28:44.785911 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:44.785914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:44.785923 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:28:44.785948 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:44.785949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:44.785962 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:44.786020 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:44.786236 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.786811 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.787797 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.788276 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.788364 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.788374 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:44.788522 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:28:44.788555 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260620 14:28:44.788982 14047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbc3493f8424fdaad7ca13f4afedd23" candidate_uuid: "6115885c5692438c88137fac7d081af1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" is_pre_election: true
I20260620 14:28:44.788983 13779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbc3493f8424fdaad7ca13f4afedd23" candidate_uuid: "6115885c5692438c88137fac7d081af1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59cc386a6f484094b8b40f15b9977a65" is_pre_election: true
I20260620 14:28:44.789144 14047 raft_consensus.cc:2454] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6115885c5692438c88137fac7d081af1 in term 0.
I20260620 14:28:44.789144 13779 raft_consensus.cc:2454] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6115885c5692438c88137fac7d081af1 in term 0.
I20260620 14:28:44.789386 13849 leader_election.cc:302] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cc386a6f484094b8b40f15b9977a65, 6115885c5692438c88137fac7d081af1; no voters:
I20260620 14:28:44.789531 14116 raft_consensus.cc:2781] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:28:44.789590 14116 raft_consensus.cc:482] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:28:44.789613 14116 raft_consensus.cc:3037] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:44.790320 14116 raft_consensus.cc:504] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } } peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260620 14:28:44.790539 14116 leader_election.cc:288] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [CANDIDATE]: Term 1 election: Requested vote from peers 59cc386a6f484094b8b40f15b9977a65 (127.0.98.1:44759), bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199)
I20260620 14:28:44.790686 13779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbc3493f8424fdaad7ca13f4afedd23" candidate_uuid: "6115885c5692438c88137fac7d081af1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59cc386a6f484094b8b40f15b9977a65"
I20260620 14:28:44.790709 14047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbc3493f8424fdaad7ca13f4afedd23" candidate_uuid: "6115885c5692438c88137fac7d081af1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9"
I20260620 14:28:44.790771 13779 raft_consensus.cc:3037] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:44.790796 14047 raft_consensus.cc:3037] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:44.791507 14047 raft_consensus.cc:2454] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6115885c5692438c88137fac7d081af1 in term 1.
I20260620 14:28:44.791558 13779 raft_consensus.cc:2454] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6115885c5692438c88137fac7d081af1 in term 1.
I20260620 14:28:44.791745 13847 leader_election.cc:302] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6115885c5692438c88137fac7d081af1, bda0b3ebe98d4d78915c4eb59126fcd9; no voters:
I20260620 14:28:44.791850 14116 raft_consensus.cc:2781] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:44.791996 14116 raft_consensus.cc:686] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 1 LEADER]: Becoming Leader. State: Replica: 6115885c5692438c88137fac7d081af1, State: Running, Role: LEADER
I20260620 14:28:44.792124 14116 consensus_queue.cc:237] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } } peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260620 14:28:44.793015 13633 catalog_manager.cc:5495] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6115885c5692438c88137fac7d081af1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6115885c5692438c88137fac7d081af1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } health_report { overall_health: HEALTHY } } }
I20260620 14:28:44.850020 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.847486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.0.98.2:36991)
0620 14:28:44.847547 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:44.847549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:44.847560 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:28:44.847649 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:44.847650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:44.847672 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:44.847760 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:28:44.847929 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.848410 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.849210 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:44.849409 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:44.849493 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:44.849673 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:28:44.849776 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:28:44.849874 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:44.849902 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 14:28:44.849923 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:28:44.852030 13779 raft_consensus.cc:1261] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Refusing update from remote peer 6115885c5692438c88137fac7d081af1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:44.852054 14047 raft_consensus.cc:1261] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9 [term 1 FOLLOWER]: Refusing update from remote peer 6115885c5692438c88137fac7d081af1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:44.852411 14116 consensus_queue.cc:1035] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:44.852558 14121 consensus_queue.cc:1035] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:44.855815 14124 mvcc.cc:205] Tried to move back new op lower bound from 7298931608990453760 to 7298931608748990464. Current Snapshot: MvccSnapshot[applied={T|T < 7298931608990453760 or (T in {7298931608990453760})}]
I20260620 14:28:44.857667 13633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:44.857764 13633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:44.858611 13633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Sending DeleteTablet for 3 replicas of tablet 3fbc3493f8424fdaad7ca13f4afedd23
I20260620 14:28:44.858953 13759 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbc3493f8424fdaad7ca13f4afedd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33872
I20260620 14:28:44.858997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 13696
I20260620 14:28:44.858991 14027 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbc3493f8424fdaad7ca13f4afedd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45378
I20260620 14:28:44.859679 13893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbc3493f8424fdaad7ca13f4afedd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48884
I20260620 14:28:44.859911 14134 tablet_replica.cc:330] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9: stopping tablet replica
I20260620 14:28:44.860033 14134 raft_consensus.cc:2229] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:44.860138 14134 raft_consensus.cc:2258] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:44.860370 14135 tablet_replica.cc:330] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1: stopping tablet replica
I20260620 14:28:44.860718 14134 ts_tablet_manager.cc:1905] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:44.862469 14135 raft_consensus.cc:2229] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:28:44.862648 14135 raft_consensus.cc:2258] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:44.863365 14135 ts_tablet_manager.cc:1905] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:44.863433 14134 ts_tablet_manager.cc:1918] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:44.863588 14134 log.cc:1196] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/3fbc3493f8424fdaad7ca13f4afedd23
I20260620 14:28:44.863904 14134 ts_tablet_manager.cc:1939] T 3fbc3493f8424fdaad7ca13f4afedd23 P bda0b3ebe98d4d78915c4eb59126fcd9: Deleting consensus metadata
I20260620 14:28:44.864389 13619 catalog_manager.cc:4841] TS bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199): tablet 3fbc3493f8424fdaad7ca13f4afedd23 (table test-table [id=e4dd8887a2cb49fc810bbe8596ad2733]) successfully deleted
I20260620 14:28:44.865046 14135 ts_tablet_manager.cc:1918] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:44.865123 14135 log.cc:1196] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/3fbc3493f8424fdaad7ca13f4afedd23
I20260620 14:28:44.865388 14135 ts_tablet_manager.cc:1939] T 3fbc3493f8424fdaad7ca13f4afedd23 P 6115885c5692438c88137fac7d081af1: Deleting consensus metadata
I20260620 14:28:44.865811 13618 catalog_manager.cc:4841] TS 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991): tablet 3fbc3493f8424fdaad7ca13f4afedd23 (table test-table [id=e4dd8887a2cb49fc810bbe8596ad2733]) successfully deleted
W20260620 14:28:44.866657 13621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44759 (error 108)
I20260620 14:28:44.866882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 13830
I20260620 14:28:44.868281 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:44.868162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44759 (local address 127.0.0.1:33886)
0620 14:28:44.868200 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:44.868240 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 14:28:44.868412 13621 catalog_manager.cc:4584] TS 59cc386a6f484094b8b40f15b9977a65 (127.0.98.1:44759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fbc3493f8424fdaad7ca13f4afedd23: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44759: connect: Connection refused (error 111)
I20260620 14:28:44.871917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 13964
I20260620 14:28:44.876624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 13602
I20260620 14:28:44.882289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46071
--webserver_interface=127.0.98.62
--webserver_port=36441
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:44.967208 14137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:44.967489 14137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:44.967531 14137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:44.968720 14137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:44.968801 14137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:44.968825 14137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:44.968847 14137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:44.968878 14137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:44.970407 14137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:44.970701 14137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:44.970956 14137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:44.980268 14143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:44.980260 14146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:44.980353 14137 server_base.cc:1034] running on GCE node
W20260620 14:28:44.980288 14144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:44.980813 14137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:44.981237 14137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:44.982415 14137 hybrid_clock.cc:648] HybridClock initialized: now 1781965724982386 us; error 39 us; skew 500 ppm
Jun 20 14:28:44 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:44.984908 14137 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:44.985414 14137 webserver.cc:460] Webserver started at http://127.0.98.62:36441/ using document root <none> and password file <none>
I20260620 14:28:44.985668 14137 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:44.985754 14137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:44.987447 14137 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:44.988390 14153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:44.988641 14137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:44.988766 14137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc"
format_stamp: "Formatted at 2026-06-20 14:28:43 on dist-test-slave-hfc3"
server_key: "53edfccede37b18677e836ea12bc4376"
server_key_iv: "d5dff4c3a8fbd1a9af84a714c74f0cfd"
server_key_version: "encryptionkey@0"
I20260620 14:28:44.989207 14137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:45.024719 14137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:45.026969 14137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:45.027182 14137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:45.031874 14137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46071
I20260620 14:28:45.031934 14205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46071 every 8 connection(s)
I20260620 14:28:45.032382 14137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:45.034030 14206 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:28:45.034751 14206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Bootstrap starting.
I20260620 14:28:45.038159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 14137
I20260620 14:28:45.038542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44759
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46071
--webserver_port=40879
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:45.044041 14206 log.cc:824] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:45.046139 14206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:45.046499 14206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Bootstrap complete.
I20260620 14:28:45.048863 14206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260620 14:28:45.049116 14206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbe4b02cb244bf5a4b6806a7d3d95fc, State: Initialized, Role: FOLLOWER
I20260620 14:28:45.049312 14206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260620 14:28:45.049422 14206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:45.049485 14206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:45.049532 14206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:28:45.050997 14206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260620 14:28:45.051134 14206 leader_election.cc:302] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dbe4b02cb244bf5a4b6806a7d3d95fc; no voters:
I20260620 14:28:45.051353 14206 leader_election.cc:288] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:28:45.051390 14210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:28:45.051506 14210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 2 LEADER]: Becoming Leader. State: Replica: 1dbe4b02cb244bf5a4b6806a7d3d95fc, State: Running, Role: LEADER
I20260620 14:28:45.051646 14206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:45.051658 14210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260620 14:28:45.052070 14211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } }
I20260620 14:28:45.052093 14212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dbe4b02cb244bf5a4b6806a7d3d95fc. Latest consensus state: current_term: 2 leader_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } }
I20260620 14:28:45.052145 14211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:45.052176 14212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:45.052601 14215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:45.053514 14215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:45.053627 14215 catalog_manager.cc:1259] Loaded cluster ID: 894566a759cf4fd9859074cbbf675e43
I20260620 14:28:45.053709 14215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:45.054105 14215 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 14:28:45.057549 14215 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 14:28:45.063654 14215 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:45.063964 14215 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Loaded TSK: 0
I20260620 14:28:45.064226 14215 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:28:45.144258 14208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:45.144505 14208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:45.144555 14208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:45.145802 14208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:45.145881 14208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:45.145929 14208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:28:45.147593 14208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40879
--tserver_master_addrs=127.0.98.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:45.147890 14208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:45.148140 14208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:45.154701 14234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:45.154773 14236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:45.154731 14233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:45.155102 14208 server_base.cc:1034] running on GCE node
I20260620 14:28:45.155344 14208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:45.155614 14208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:45.156790 14208 hybrid_clock.cc:648] HybridClock initialized: now 1781965725156775 us; error 34 us; skew 500 ppm
Jun 20 14:28:45 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:45.159437 14208 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:28:45.159982 14208 webserver.cc:460] Webserver started at http://127.0.98.1:40879/ using document root <none> and password file <none>
I20260620 14:28:45.160223 14208 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:45.160282 14208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:45.161696 14208 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:28:45.162436 14243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:45.162616 14208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:45.162726 14208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "59cc386a6f484094b8b40f15b9977a65"
format_stamp: "Formatted at 2026-06-20 14:28:43 on dist-test-slave-hfc3"
server_key: "dec28a6ac82a0219ae4ef4770907db6d"
server_key_iv: "95bf7977b675e7c418acd17574e84e62"
server_key_version: "encryptionkey@0"
I20260620 14:28:45.163096 14208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:45.185351 14208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:45.187876 14208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:45.187996 14208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:45.188310 14208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:45.188802 14250 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:28:45.190289 14208 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:28:45.190384 14208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:28:45.190421 14208 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:28:45.191157 14208 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:28:45.191205 14208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:28:45.191248 14250 tablet_bootstrap.cc:492] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Bootstrap starting.
I20260620 14:28:45.199519 14208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44759
I20260620 14:28:45.200107 14208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:28:45.200634 14357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44759 every 8 connection(s)
I20260620 14:28:45.203132 14250 log.cc:824] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:45.205065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 14208
I20260620 14:28:45.205353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36991
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46071
--webserver_port=39701
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:45.205651 14250 tablet_bootstrap.cc:492] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:45.205948 14250 tablet_bootstrap.cc:492] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Bootstrap complete.
I20260620 14:28:45.207151 14250 ts_tablet_manager.cc:1397] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20260620 14:28:45.208424 14250 raft_consensus.cc:348] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } } peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260620 14:28:45.208743 14250 raft_consensus.cc:729] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cc386a6f484094b8b40f15b9977a65, State: Initialized, Role: FOLLOWER
I20260620 14:28:45.208928 14250 consensus_queue.cc:260] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } } peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260620 14:28:45.209225 14250 ts_tablet_manager.cc:1428] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
Jun 20 14:28:45 dist-test-slave-hfc3 krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:45.215871 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.201302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45119 (local address 127.0.98.62:46071)
0620 14:28:45.202510 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:45.202521 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:45.202564 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:28:45.202724 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:45.202730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:45.202790 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:45.202934 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:45.203787 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.205401 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.208108 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.208748 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.208868 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.211305 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:45.211309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:45.211310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:45.211323 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:45.213598 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:45.214176 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:45.214179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:45.214180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:45.214314 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:45.214651 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:45.214652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:45.214652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:45.214967 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:45.215061 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:45.215501 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 14:28:45.215657 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":98,"threads_started":1}
I20260620 14:28:45.215899 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.200895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:45119)
0620 14:28:45.201526 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:45.201578 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:45.202246 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:45.203023 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:45.203027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:45.203406 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:45.203642 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.203648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.206465 (+ 2817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.206466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.207928 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.207931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.208963 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.208965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.209103 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.209716 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:45.209732 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:45.211181 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:45.213703 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:45.213712 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:45.213714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:45.214075 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:45.214412 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:45.214413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:45.214413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:45.214567 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:45.215098 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:45.215106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:45.215291 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 14:28:45.215425 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 14:28:45.215632 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":2,"thread_start_us":101,"threads_started":1}
I20260620 14:28:45.216722 14358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46071
I20260620 14:28:45.216825 14358 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:45.217074 14358 heartbeater.cc:511] Master 127.0.98.62:46071 requested a full tablet report, sending...
I20260620 14:28:45.217937 14170 ts_manager.cc:194] Registered new tserver with Master: 59cc386a6f484094b8b40f15b9977a65 (127.0.98.1:44759)
I20260620 14:28:45.219300 14170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45119
I20260620 14:28:45.220257 14358 heartbeater.cc:503] Master 127.0.98.62:46071 was elected leader, sending a full tablet report...
Jun 20 14:28:45 dist-test-slave-hfc3 krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:28:45.229365 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.218661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44759 (local address 127.0.0.1:33888)
0620 14:28:45.219122 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:45.219161 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:45.219229 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:45.219730 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:45.219734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:45.220880 (+ 1146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:45.220942 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.220945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.222252 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.222253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.223103 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.223106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.223652 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.223656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.223740 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.224385 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:45.224390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:45.225940 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:45.228211 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:45.228218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:45.228220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:45.228441 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:45.228671 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:45.228672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:45.228672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:45.228731 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:45.229050 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:45.229056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:45.229104 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:28:45.229177 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:28:45.229282 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":103,"threads_started":1}
I20260620 14:28:45.229435 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.218937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.0.98.1:44759)
0620 14:28:45.219355 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:45.219359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:45.219455 (+ 96us) server_negotiation.cc:407] Connection header received
0620 14:28:45.219495 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:45.219497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:45.219512 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:45.219665 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:45.221048 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.222149 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.223215 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.223546 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.223642 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.226031 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:45.226046 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:45.226046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:45.226055 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:45.228113 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:45.228529 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:45.228532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:45.228533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:45.228599 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:45.228798 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:45.228799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:45.228799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:45.228982 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:45.229069 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:45.229233 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:28:45.229335 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":94,"threads_started":1}
I20260620 14:28:45.230050 14289 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbc3493f8424fdaad7ca13f4afedd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33888
I20260620 14:28:45.230707 14367 tablet_replica.cc:330] stopping tablet replica
I20260620 14:28:45.230846 14367 raft_consensus.cc:2229] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:45.230952 14367 raft_consensus.cc:2258] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:45.231321 14367 ts_tablet_manager.cc:1905] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:45.233433 14367 ts_tablet_manager.cc:1918] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:45.233541 14367 log.cc:1196] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal/wals/3fbc3493f8424fdaad7ca13f4afedd23
I20260620 14:28:45.233793 14367 ts_tablet_manager.cc:1939] T 3fbc3493f8424fdaad7ca13f4afedd23 P 59cc386a6f484094b8b40f15b9977a65: Deleting consensus metadata
I20260620 14:28:45.234452 14157 catalog_manager.cc:4841] TS 59cc386a6f484094b8b40f15b9977a65 (127.0.98.1:44759): tablet 3fbc3493f8424fdaad7ca13f4afedd23 (table test-table [id=e4dd8887a2cb49fc810bbe8596ad2733]) successfully deleted
W20260620 14:28:45.298350 14363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:45.298588 14363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:45.298612 14363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:45.299790 14363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:45.299836 14363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:45.299863 14363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:28:45.301362 14363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39701
--tserver_master_addrs=127.0.98.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:45.301625 14363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:45.301868 14363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:45.306841 14372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:45.306833 14375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:45.306831 14373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:45.307107 14363 server_base.cc:1034] running on GCE node
I20260620 14:28:45.307334 14363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:45.307600 14363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:45.308769 14363 hybrid_clock.cc:648] HybridClock initialized: now 1781965725308750 us; error 62 us; skew 500 ppm
Jun 20 14:28:45 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:45.311448 14363 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:28:45.311933 14363 webserver.cc:460] Webserver started at http://127.0.98.2:39701/ using document root <none> and password file <none>
I20260620 14:28:45.312124 14363 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:45.312175 14363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:45.313521 14363 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:45.314126 14382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:45.314323 14363 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:45.314474 14363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "6115885c5692438c88137fac7d081af1"
format_stamp: "Formatted at 2026-06-20 14:28:44 on dist-test-slave-hfc3"
server_key: "25d4c9c217632634df02f9d600fea3dc"
server_key_iv: "4056189b8d6b9d2973009681ff8edf94"
server_key_version: "encryptionkey@0"
I20260620 14:28:45.314842 14363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:45.342744 14363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:45.344962 14363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:45.345113 14363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:45.345371 14363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:45.345777 14363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:45.345839 14363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:45.345876 14363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:45.345902 14363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:45.351745 14363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36991
I20260620 14:28:45.351830 14495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36991 every 8 connection(s)
I20260620 14:28:45.352259 14363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:28:45.355907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 14363
I20260620 14:28:45.356174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40199
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46071
--webserver_port=37827
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:28:45 dist-test-slave-hfc3 krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:45.362603 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.353058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52623 (local address 127.0.98.62:46071)
0620 14:28:45.353105 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:45.353107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:45.353428 (+ 321us) server_negotiation.cc:407] Connection header received
0620 14:28:45.353909 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:45.353910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:45.353923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:45.353978 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:45.355000 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.355488 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.356723 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.357057 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.357132 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.359618 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:45.359619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:45.359620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:45.359631 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:45.361122 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:45.361554 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:45.361556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:45.361556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:45.361631 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:45.361893 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:45.361894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:45.361894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:45.362025 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:45.362105 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:45.362405 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 14:28:45.362499 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:28:45.362774 14498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.353089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:52623)
0620 14:28:45.353297 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:45.353334 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:45.353821 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:45.354051 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:45.354055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:45.354713 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:45.354909 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.354914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.355588 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.355589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.356636 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.356639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.357145 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.357146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.357248 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.357850 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:45.357861 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:45.359500 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:45.361222 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:45.361230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:45.361232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:45.361461 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:45.361708 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:45.361709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:45.361709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:45.361818 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:45.362099 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:45.362105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:45.362246 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 14:28:45.362377 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:28:45.362538 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260620 14:28:45.363263 14496 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46071
I20260620 14:28:45.363360 14496 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:45.363565 14496 heartbeater.cc:511] Master 127.0.98.62:46071 requested a full tablet report, sending...
I20260620 14:28:45.363965 14170 ts_manager.cc:194] Registered new tserver with Master: 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991)
I20260620 14:28:45.364537 14170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52623
W20260620 14:28:45.439576 14499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:45.439788 14499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:45.439811 14499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:45.440877 14499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:45.440938 14499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:45.440966 14499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:28:45.442369 14499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37827
--tserver_master_addrs=127.0.98.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:45.442557 14499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:45.442726 14499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:45.447515 14507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:45.447582 14504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:45.447540 14505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:45.447775 14499 server_base.cc:1034] running on GCE node
I20260620 14:28:45.447983 14499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:45.448228 14499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:45.449407 14499 hybrid_clock.cc:648] HybridClock initialized: now 1781965725449398 us; error 28 us; skew 500 ppm
Jun 20 14:28:45 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:45.452042 14499 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:28:45.452607 14499 webserver.cc:460] Webserver started at http://127.0.98.3:37827/ using document root <none> and password file <none>
I20260620 14:28:45.452800 14499 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:45.452883 14499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:45.454280 14499 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:45.454913 14514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:45.455088 14499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:45.455196 14499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "bda0b3ebe98d4d78915c4eb59126fcd9"
format_stamp: "Formatted at 2026-06-20 14:28:44 on dist-test-slave-hfc3"
server_key: "f568411950b4aa164147656f7b7b2353"
server_key_iv: "822509907c513478a81c5d2fe9803187"
server_key_version: "encryptionkey@0"
I20260620 14:28:45.455560 14499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:45.476716 14499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:45.479360 14499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:45.479537 14499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:45.479790 14499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:45.480141 14499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:45.480181 14499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:45.480206 14499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:45.480255 14499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:45.486289 14499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40199
I20260620 14:28:45.486379 14627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40199 every 8 connection(s)
I20260620 14:28:45.486853 14499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:28:45.492048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 14499
Jun 20 14:28:45 dist-test-slave-hfc3 krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:45.496707 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.487654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52901 (local address 127.0.98.62:46071)
0620 14:28:45.487703 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:45.487705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:45.487898 (+ 193us) server_negotiation.cc:407] Connection header received
0620 14:28:45.488439 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:45.488441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:45.488453 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:45.488507 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:45.489490 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.489984 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.491013 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.491219 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.491327 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.493820 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:45.493821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:45.493821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:45.493827 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:45.495289 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:45.495709 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:45.495710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:45.495711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:45.495768 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:45.496027 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:45.496028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:45.496028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:45.496134 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:45.496221 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:45.496514 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 14:28:45.496609 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:28:45.496935 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.487613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:52901)
0620 14:28:45.487759 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:45.487804 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:45.488325 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:45.488580 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:45.488584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:45.489179 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:45.489378 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.489383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.490082 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.490084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.490915 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.490917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.491334 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.491336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.491461 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.492032 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:45.492040 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:45.493733 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:45.495375 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:45.495382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:45.495384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:45.495604 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:45.495837 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:45.495838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:45.495838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:45.495950 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:45.496211 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:45.496217 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:45.496366 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 14:28:45.496479 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:28:45.496681 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260620 14:28:45.497462 14628 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46071
I20260620 14:28:45.497568 14628 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:45.497773 14628 heartbeater.cc:511] Master 127.0.98.62:46071 requested a full tablet report, sending...
I20260620 14:28:45.498267 14170 ts_manager.cc:194] Registered new tserver with Master: bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199)
I20260620 14:28:45.498689 14170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52901
I20260620 14:28:45.500061 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.492497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.0.98.62:46071)
0620 14:28:45.492619 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:45.492621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:45.493337 (+ 716us) server_negotiation.cc:407] Connection header received
0620 14:28:45.493391 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:45.493392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:45.493405 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:45.493484 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:45.494193 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.494894 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.495693 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.495891 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.495978 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.497431 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:45.497433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:45.497433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:45.497441 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:45.499089 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:45.499418 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:45.499419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:45.499419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:45.499482 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:45.499657 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:45.499657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:45.499658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:45.499748 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:45.499856 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:45.499915 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:28:45.499975 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260620 14:28:45.500825 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:28:45.503800 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.501202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.0.98.62:46071)
0620 14:28:45.501263 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:45.501265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:45.501363 (+ 98us) server_negotiation.cc:407] Connection header received
0620 14:28:45.501439 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:45.501439 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:45.501473 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:45.501567 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:28:45.501667 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.502130 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.502974 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.503189 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.503277 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.503464 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:28:45.503569 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:28:45.503662 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:45.503703 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 14:28:45.503726 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:28:45.504087 14170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:28:45.505692 14170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:28:45.507879 14289 tablet_service.cc:1467] Processing CreateTablet for tablet ad625f3e60b54ef2beafaf8f38a3666e (DEFAULT_TABLE table=test-table [id=b8a521bf5df84a77b936a4d357ea17d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:45.508116 14289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad625f3e60b54ef2beafaf8f38a3666e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:45.509547 14250 tablet_bootstrap.cc:492] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: Bootstrap starting.
I20260620 14:28:45.509871 14250 tablet_bootstrap.cc:654] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:45.510520 14250 tablet_bootstrap.cc:492] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: No bootstrap required, opened a new log
I20260620 14:28:45.510578 14250 ts_tablet_manager.cc:1397] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:28:45.510921 14250 raft_consensus.cc:348] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:45.511013 14250 raft_consensus.cc:374] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:45.511065 14250 raft_consensus.cc:729] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cc386a6f484094b8b40f15b9977a65, State: Initialized, Role: FOLLOWER
I20260620 14:28:45.511107 14250 consensus_queue.cc:260] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:45.511196 14250 ts_tablet_manager.cc:1428] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:28:45.512516 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.507665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40199 (local address 127.0.0.1:45388)
0620 14:28:45.508292 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:45.508303 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:45.508369 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:45.508560 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:45.508561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:45.508563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:45.508611 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.508613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.510019 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.510020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.511145 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.511149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.512247 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.512248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.512313 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.512328 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:28:45.512395 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:28:45.512439 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":522,"thread_start_us":59,"threads_started":1}
I20260620 14:28:45.512540 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.507789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45388 (local address 127.0.98.3:40199)
0620 14:28:45.508005 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:45.508010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:45.508385 (+ 375us) server_negotiation.cc:407] Connection header received
0620 14:28:45.508429 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:45.508431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:45.508445 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:45.508783 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:45.508850 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.509898 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.511266 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.512009 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.512061 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.512126 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:45.512425 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 14:28:45.512466 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":109,"threads_started":1}
I20260620 14:28:45.512655 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.507665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36991 (local address 127.0.0.1:48890)
0620 14:28:45.507702 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:45.507713 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:45.507752 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:45.508634 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:45.508635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:45.508637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:45.508683 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.508684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.510047 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.510048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.511245 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.511248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.512387 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.512388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.512462 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.512467 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:28:45.512534 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:28:45.512582 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":57}
I20260620 14:28:45.512669 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.508040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.0.98.2:36991)
0620 14:28:45.508218 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:45.508222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:45.508234 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:28:45.508279 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:45.508281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:45.508295 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:45.508408 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:45.508742 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.509907 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.511508 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.512289 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.512385 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.512443 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:45.512555 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:28:45.512590 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260620 14:28:45.513494 14562 tablet_service.cc:1467] Processing CreateTablet for tablet ad625f3e60b54ef2beafaf8f38a3666e (DEFAULT_TABLE table=test-table [id=b8a521bf5df84a77b936a4d357ea17d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:45.513556 14430 tablet_service.cc:1467] Processing CreateTablet for tablet ad625f3e60b54ef2beafaf8f38a3666e (DEFAULT_TABLE table=test-table [id=b8a521bf5df84a77b936a4d357ea17d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:45.513856 14562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad625f3e60b54ef2beafaf8f38a3666e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:45.513871 14430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad625f3e60b54ef2beafaf8f38a3666e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:45.516750 14638 tablet_bootstrap.cc:492] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9: Bootstrap starting.
I20260620 14:28:45.516793 14639 tablet_bootstrap.cc:492] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1: Bootstrap starting.
I20260620 14:28:45.517356 14638 tablet_bootstrap.cc:654] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:45.517508 14639 tablet_bootstrap.cc:654] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:45.517977 14638 log.cc:824] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:45.518008 14639 log.cc:824] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:45.518884 14638 tablet_bootstrap.cc:492] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9: No bootstrap required, opened a new log
I20260620 14:28:45.518957 14638 ts_tablet_manager.cc:1397] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:45.518999 14639 tablet_bootstrap.cc:492] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1: No bootstrap required, opened a new log
I20260620 14:28:45.519070 14639 ts_tablet_manager.cc:1397] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:45.520387 14639 raft_consensus.cc:348] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:45.520370 14638 raft_consensus.cc:348] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:45.520545 14638 raft_consensus.cc:374] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:45.520547 14639 raft_consensus.cc:374] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:45.520596 14639 raft_consensus.cc:729] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6115885c5692438c88137fac7d081af1, State: Initialized, Role: FOLLOWER
I20260620 14:28:45.520598 14638 raft_consensus.cc:729] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda0b3ebe98d4d78915c4eb59126fcd9, State: Initialized, Role: FOLLOWER
I20260620 14:28:45.520718 14639 consensus_queue.cc:260] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:45.520718 14638 consensus_queue.cc:260] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:45.521071 14496 heartbeater.cc:503] Master 127.0.98.62:46071 was elected leader, sending a full tablet report...
I20260620 14:28:45.521070 14638 ts_tablet_manager.cc:1428] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:45.521076 14628 heartbeater.cc:503] Master 127.0.98.62:46071 was elected leader, sending a full tablet report...
I20260620 14:28:45.521365 14639 ts_tablet_manager.cc:1428] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260620 14:28:45.602993 14497 tablet.cc:2369] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:28:45.620457 14364 raft_consensus.cc:482] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:45.620631 14364 raft_consensus.cc:504] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:45.621062 14364 leader_election.cc:288] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199), 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991)
I20260620 14:28:45.624984 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.621245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40199 (local address 127.0.98.1:56133)
0620 14:28:45.621356 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:45.621370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:45.621436 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:45.621645 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:45.621647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:45.621648 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:45.621724 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.621726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.622692 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.622693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.623969 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.623972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.624666 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.624668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.624753 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.624784 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:28:45.624852 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:28:45.624901 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 14:28:45.624995 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.621415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56133 (local address 127.0.98.3:40199)
0620 14:28:45.621497 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:45.621499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:45.621510 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:28:45.621541 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:45.621542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:45.621554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:45.621630 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:45.621808 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.622597 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.624071 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.624571 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.624623 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.624632 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:45.624884 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 14:28:45.624924 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:28:45.625180 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.621245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36991 (local address 127.0.98.1:34077)
0620 14:28:45.621736 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:45.621748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:45.621831 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:45.622060 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:45.622061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:45.622063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:45.622147 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.622149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.623032 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.623033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.624264 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:45.624267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.624908 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.624910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:45.624974 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.624978 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:28:45.625041 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:28:45.625081 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":105,"thread_start_us":107,"threads_started":1}
I20260620 14:28:45.625198 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.621329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34077 (local address 127.0.98.2:36991)
0620 14:28:45.621543 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:45.621545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:45.621802 (+ 257us) server_negotiation.cc:407] Connection header received
0620 14:28:45.621920 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:45.621921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:45.621960 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:45.621993 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:45.622255 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.622935 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.624341 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.624816 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.624879 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.624888 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:45.625067 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:28:45.625113 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260620 14:28:45.625505 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad625f3e60b54ef2beafaf8f38a3666e" candidate_uuid: "59cc386a6f484094b8b40f15b9977a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" is_pre_election: true
I20260620 14:28:45.625684 14582 raft_consensus.cc:2454] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59cc386a6f484094b8b40f15b9977a65 in term 0.
I20260620 14:28:45.625680 14450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad625f3e60b54ef2beafaf8f38a3666e" candidate_uuid: "59cc386a6f484094b8b40f15b9977a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6115885c5692438c88137fac7d081af1" is_pre_election: true
I20260620 14:28:45.625851 14450 raft_consensus.cc:2454] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59cc386a6f484094b8b40f15b9977a65 in term 0.
I20260620 14:28:45.625955 14245 leader_election.cc:302] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cc386a6f484094b8b40f15b9977a65, bda0b3ebe98d4d78915c4eb59126fcd9; no voters:
I20260620 14:28:45.626087 14364 raft_consensus.cc:2781] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:28:45.626166 14364 raft_consensus.cc:482] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:28:45.626199 14364 raft_consensus.cc:3037] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:45.627082 14364 raft_consensus.cc:504] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:45.627241 14364 leader_election.cc:288] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 1 election: Requested vote from peers bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199), 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991)
I20260620 14:28:45.627384 14450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad625f3e60b54ef2beafaf8f38a3666e" candidate_uuid: "59cc386a6f484094b8b40f15b9977a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6115885c5692438c88137fac7d081af1"
I20260620 14:28:45.627380 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad625f3e60b54ef2beafaf8f38a3666e" candidate_uuid: "59cc386a6f484094b8b40f15b9977a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9"
I20260620 14:28:45.627463 14450 raft_consensus.cc:3037] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:45.627483 14582 raft_consensus.cc:3037] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:45.628221 14582 raft_consensus.cc:2454] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59cc386a6f484094b8b40f15b9977a65 in term 1.
I20260620 14:28:45.628230 14450 raft_consensus.cc:2454] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59cc386a6f484094b8b40f15b9977a65 in term 1.
I20260620 14:28:45.628405 14244 leader_election.cc:302] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cc386a6f484094b8b40f15b9977a65, 6115885c5692438c88137fac7d081af1; no voters:
I20260620 14:28:45.628500 14364 raft_consensus.cc:2781] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:45.628659 14364 raft_consensus.cc:686] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 LEADER]: Becoming Leader. State: Replica: 59cc386a6f484094b8b40f15b9977a65, State: Running, Role: LEADER
I20260620 14:28:45.628720 14364 consensus_queue.cc:237] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:45.629616 14169 catalog_manager.cc:5495] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59cc386a6f484094b8b40f15b9977a65 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "59cc386a6f484094b8b40f15b9977a65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } health_report { overall_health: HEALTHY } } }
I20260620 14:28:45.660720 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.658212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.0.98.1:44759)
0620 14:28:45.658279 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:45.658281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:45.658290 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:28:45.658385 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:45.658388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:45.658413 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:45.658491 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:28:45.658633 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.659127 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.659938 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:45.660137 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:45.660191 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:45.660327 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:28:45.660431 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:28:45.660487 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:45.660587 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:28:45.660607 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:28:45.662451 14450 raft_consensus.cc:1261] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1 [term 1 FOLLOWER]: Refusing update from remote peer 59cc386a6f484094b8b40f15b9977a65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:45.662472 14582 raft_consensus.cc:1261] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9 [term 1 FOLLOWER]: Refusing update from remote peer 59cc386a6f484094b8b40f15b9977a65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:45.662858 14645 consensus_queue.cc:1035] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:45.662971 14364 consensus_queue.cc:1035] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:45.665711 14649 mvcc.cc:205] Tried to move back new op lower bound from 7298931612309573632 to 7298931612175220736. Current Snapshot: MvccSnapshot[applied={T|T < 7298931612309573632}]
I20260620 14:28:45.666152 14650 mvcc.cc:205] Tried to move back new op lower bound from 7298931612309573632 to 7298931612175220736. Current Snapshot: MvccSnapshot[applied={T|T < 7298931612309573632}]
I20260620 14:28:45.668156 14170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:45.668262 14170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:45.669041 14170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Sending DeleteTablet for 3 replicas of tablet ad625f3e60b54ef2beafaf8f38a3666e
I20260620 14:28:45.669397 14562 tablet_service.cc:1514] Processing DeleteTablet for tablet ad625f3e60b54ef2beafaf8f38a3666e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45388
I20260620 14:28:45.669646 14657 tablet_replica.cc:330] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9: stopping tablet replica
I20260620 14:28:45.669669 14289 tablet_service.cc:1514] Processing DeleteTablet for tablet ad625f3e60b54ef2beafaf8f38a3666e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33888
I20260620 14:28:45.669754 14367 tablet_replica.cc:330] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: stopping tablet replica
I20260620 14:28:45.669759 14657 raft_consensus.cc:2229] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:45.669796 14367 raft_consensus.cc:2229] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:28:45.669860 14657 raft_consensus.cc:2258] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:45.669870 14367 raft_consensus.cc:2258] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:45.669847 14430 tablet_service.cc:1514] Processing DeleteTablet for tablet ad625f3e60b54ef2beafaf8f38a3666e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48890
I20260620 14:28:45.670101 14658 tablet_replica.cc:330] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1: stopping tablet replica
I20260620 14:28:45.670305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 14208
I20260620 14:28:45.670302 14658 raft_consensus.cc:2229] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:45.670481 14367 ts_tablet_manager.cc:1905] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:45.671105 14657 ts_tablet_manager.cc:1905] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:45.671082 14658 raft_consensus.cc:2258] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:45.671557 14658 ts_tablet_manager.cc:1905] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:45.673095 14658 ts_tablet_manager.cc:1918] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:45.673110 14657 ts_tablet_manager.cc:1918] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:45.673229 14657 log.cc:1196] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/ad625f3e60b54ef2beafaf8f38a3666e
I20260620 14:28:45.673233 14658 log.cc:1196] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/ad625f3e60b54ef2beafaf8f38a3666e
I20260620 14:28:45.673520 14658 ts_tablet_manager.cc:1939] T ad625f3e60b54ef2beafaf8f38a3666e P 6115885c5692438c88137fac7d081af1: Deleting consensus metadata
I20260620 14:28:45.673544 14657 ts_tablet_manager.cc:1939] T ad625f3e60b54ef2beafaf8f38a3666e P bda0b3ebe98d4d78915c4eb59126fcd9: Deleting consensus metadata
I20260620 14:28:45.673971 14155 catalog_manager.cc:4841] TS bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199): tablet ad625f3e60b54ef2beafaf8f38a3666e (table test-table [id=b8a521bf5df84a77b936a4d357ea17d1]) successfully deleted
I20260620 14:28:45.674083 14154 catalog_manager.cc:4841] TS 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991): tablet ad625f3e60b54ef2beafaf8f38a3666e (table test-table [id=b8a521bf5df84a77b936a4d357ea17d1]) successfully deleted
W20260620 14:28:45.677205 14157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44759 (error 108)
I20260620 14:28:45.677500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 14363
I20260620 14:28:45.677731 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:45.677612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44759 (local address 127.0.0.1:33906)
0620 14:28:45.677659 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:45.677693 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:28:45.677836 14157 catalog_manager.cc:4584] TS 59cc386a6f484094b8b40f15b9977a65 (127.0.98.1:44759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad625f3e60b54ef2beafaf8f38a3666e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44759: connect: Connection refused (error 111)
I20260620 14:28:45.683143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 14499
I20260620 14:28:45.687999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 14137
I20260620 14:28:45.693786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46071
--webserver_interface=127.0.98.62
--webserver_port=36441
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:45.780612 14660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:45.780969 14660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:45.781054 14660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:45.783344 14660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:45.783409 14660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:45.783437 14660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:45.783497 14660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:45.783529 14660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:45.785660 14660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:45.785887 14660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:45.786079 14660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:45.791181 14666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:45.791194 14668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:45.791191 14665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:45.791538 14660 server_base.cc:1034] running on GCE node
I20260620 14:28:45.791826 14660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:45.792111 14660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:45.793272 14660 hybrid_clock.cc:648] HybridClock initialized: now 1781965725793260 us; error 33 us; skew 500 ppm
Jun 20 14:28:45 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:45.795976 14660 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:45.796545 14660 webserver.cc:460] Webserver started at http://127.0.98.62:36441/ using document root <none> and password file <none>
I20260620 14:28:45.796808 14660 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:45.796885 14660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:45.798318 14660 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:28:45.799096 14675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:45.799280 14660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:45.799392 14660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc"
format_stamp: "Formatted at 2026-06-20 14:28:43 on dist-test-slave-hfc3"
server_key: "53edfccede37b18677e836ea12bc4376"
server_key_iv: "d5dff4c3a8fbd1a9af84a714c74f0cfd"
server_key_version: "encryptionkey@0"
I20260620 14:28:45.799793 14660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:45.809504 14660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:45.811703 14660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:45.811889 14660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:45.816324 14660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46071
I20260620 14:28:45.816402 14727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46071 every 8 connection(s)
I20260620 14:28:45.816939 14660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:45.818655 14728 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:28:45.818953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 14660
I20260620 14:28:45.819185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44759
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46071
--webserver_port=40879
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:45.821143 14728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Bootstrap starting.
I20260620 14:28:45.830056 14728 log.cc:824] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:45.832633 14728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:45.833168 14728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc: Bootstrap complete.
I20260620 14:28:45.835697 14728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260620 14:28:45.835940 14728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbe4b02cb244bf5a4b6806a7d3d95fc, State: Initialized, Role: FOLLOWER
I20260620 14:28:45.836113 14728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260620 14:28:45.836190 14728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:45.836242 14728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:45.836292 14728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:28:45.837340 14728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260620 14:28:45.837427 14728 leader_election.cc:302] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dbe4b02cb244bf5a4b6806a7d3d95fc; no voters:
I20260620 14:28:45.837678 14728 leader_election.cc:288] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:28:45.837711 14732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:28:45.837829 14732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [term 3 LEADER]: Becoming Leader. State: Replica: 1dbe4b02cb244bf5a4b6806a7d3d95fc, State: Running, Role: LEADER
I20260620 14:28:45.837919 14732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260620 14:28:45.837961 14728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:45.838240 14733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } }
I20260620 14:28:45.838362 14733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:45.838234 14732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dbe4b02cb244bf5a4b6806a7d3d95fc. Latest consensus state: current_term: 3 leader_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe4b02cb244bf5a4b6806a7d3d95fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } }
I20260620 14:28:45.838562 14732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbe4b02cb244bf5a4b6806a7d3d95fc [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:45.838902 14737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:45.839795 14737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:45.839919 14737 catalog_manager.cc:1259] Loaded cluster ID: 894566a759cf4fd9859074cbbf675e43
I20260620 14:28:45.840173 14737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 14:28:45.840396 14737 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3b972d29ad google::LogMessage::Fail() at ??:0
@ 0x7f3b972d588a google::LogMessage::SendToLog() at ??:0
@ 0x7f3b972d24de google::LogMessage::Flush() at ??:0
@ 0x7f3b972d3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3b9f1f80f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f3b9f2018b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3b991a00d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3b99195954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3b99ff96db start_thread at ??:0
@ 0x7f3b95dd061f clone at ??:0
W20260620 14:28:45.940070 14730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:45.940371 14730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:45.940411 14730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:45.942256 14730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:45.942374 14730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:45.942425 14730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:28:45.944670 14730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40879
--tserver_master_addrs=127.0.98.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:45.944947 14730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:45.945220 14730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:45.951043 14765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:45.951056 14767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:45.951293 14730 server_base.cc:1034] running on GCE node
W20260620 14:28:45.951354 14764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:45.951555 14730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:45.951836 14730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:45.953012 14730 hybrid_clock.cc:648] HybridClock initialized: now 1781965725952997 us; error 27 us; skew 500 ppm
Jun 20 14:28:45 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:45.955972 14730 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:28:45.956382 14730 webserver.cc:460] Webserver started at http://127.0.98.1:40879/ using document root <none> and password file <none>
I20260620 14:28:45.956569 14730 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:45.956612 14730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:45.958012 14730 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:28:45.958997 14774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:45.959252 14730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:28:45.959364 14730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "59cc386a6f484094b8b40f15b9977a65"
format_stamp: "Formatted at 2026-06-20 14:28:43 on dist-test-slave-hfc3"
server_key: "dec28a6ac82a0219ae4ef4770907db6d"
server_key_iv: "95bf7977b675e7c418acd17574e84e62"
server_key_version: "encryptionkey@0"
I20260620 14:28:45.959753 14730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:45.978915 14730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:45.981416 14730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:45.981596 14730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:45.981864 14730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:45.982389 14781 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:28:45.984382 14730 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:28:45.984454 14730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:28:45.984501 14730 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:28:45.985229 14730 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:28:45.985283 14730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:28:45.985368 14781 tablet_bootstrap.cc:492] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: Bootstrap starting.
I20260620 14:28:45.987308 14781 log.cc:824] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:45.990312 14781 tablet_bootstrap.cc:492] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:45.990667 14781 tablet_bootstrap.cc:492] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: Bootstrap complete.
I20260620 14:28:45.990821 14781 ts_tablet_manager.cc:1397] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 14:28:45.992738 14730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44759
I20260620 14:28:45.992597 14781 raft_consensus.cc:348] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:45.992946 14781 raft_consensus.cc:729] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cc386a6f484094b8b40f15b9977a65, State: Initialized, Role: FOLLOWER
I20260620 14:28:45.993170 14730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:28:45.993151 14781 consensus_queue.cc:260] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:45.993347 14889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44759 every 8 connection(s)
I20260620 14:28:45.993490 14781 ts_tablet_manager.cc:1428] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 14:28:45.995364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 14730
I20260620 14:28:45.995644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36991
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46071
--webserver_port=39701
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:46.073833 14893 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:28:45.993895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:53781)
0620 14:28:45.994058 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:45.994101 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:45.994267 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:46.073611 (+ 79344us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
W20260620 14:28:46.074100 14890 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46071 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46071: Client connection negotiation failed: client connection to 127.0.98.62:46071: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:28:46.074529 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:46.074405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:52851)
0620 14:28:46.074466 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:46.074508 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:28:46.074904 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:46.074782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:49109)
0620 14:28:46.074835 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:46.074870 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 14:28:46.075115 14890 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:28:46.080654 14894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:46.080899 14894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:46.080942 14894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:46.082134 14894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:46.082177 14894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:46.082204 14894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:28:46.084357 14894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39701
--tserver_master_addrs=127.0.98.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:46.084578 14894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:46.084767 14894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:46.089788 14899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:46.089905 14900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:46.089902 14902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:46.090394 14894 server_base.cc:1034] running on GCE node
I20260620 14:28:46.090742 14894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:46.091045 14894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:46.092226 14894 hybrid_clock.cc:648] HybridClock initialized: now 1781965726092215 us; error 41 us; skew 500 ppm
Jun 20 14:28:46 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:46.094911 14894 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:28:46.095402 14894 webserver.cc:460] Webserver started at http://127.0.98.2:39701/ using document root <none> and password file <none>
I20260620 14:28:46.095605 14894 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:46.095647 14894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:46.096944 14894 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:28:46.097522 14909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:46.097694 14894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:46.097795 14894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "6115885c5692438c88137fac7d081af1"
format_stamp: "Formatted at 2026-06-20 14:28:44 on dist-test-slave-hfc3"
server_key: "25d4c9c217632634df02f9d600fea3dc"
server_key_iv: "4056189b8d6b9d2973009681ff8edf94"
server_key_version: "encryptionkey@0"
I20260620 14:28:46.098178 14894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:46.121405 14894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:46.124045 14894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:46.124168 14894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:46.124398 14894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:46.124755 14894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:46.124799 14894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:46.124825 14894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:46.124876 14894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:46.131105 14894 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36991
I20260620 14:28:46.131157 15022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36991 every 8 connection(s)
I20260620 14:28:46.131543 14894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:28:46.131821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 14894
I20260620 14:28:46.132082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40199
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46071
--webserver_port=37827
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:46.132902 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:46.132480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:37639)
0620 14:28:46.132669 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:46.132748 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
W20260620 14:28:46.133338 15023 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46071 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46071: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
I20260620 14:28:46.133666 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:46.133554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:46657)
0620 14:28:46.133611 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:46.133645 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:28:46.133999 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:46.133918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:39369)
0620 14:28:46.133961 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:46.133984 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:28:46.134125 15023 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:28:46.215448 15026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:46.215658 15026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:46.215683 15026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:46.216771 15026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:46.216807 15026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:46.216833 15026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:28:46.218204 15026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37827
--tserver_master_addrs=127.0.98.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:28:46.218418 15026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:46.218654 15026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:28:46.223855 15026 server_base.cc:1034] running on GCE node
W20260620 14:28:46.223825 15034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:46.223825 15032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:46.223821 15031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:46.224295 15026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:46.224563 15026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:46.225715 15026 hybrid_clock.cc:648] HybridClock initialized: now 1781965726225703 us; error 54 us; skew 500 ppm
Jun 20 14:28:46 dist-test-slave-hfc3 krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:46.228430 15026 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:28:46.228973 15026 webserver.cc:460] Webserver started at http://127.0.98.3:37827/ using document root <none> and password file <none>
I20260620 14:28:46.229239 15026 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:46.229312 15026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:46.230809 15026 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:28:46.231442 15041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:46.231618 15026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:28:46.231712 15026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "bda0b3ebe98d4d78915c4eb59126fcd9"
format_stamp: "Formatted at 2026-06-20 14:28:44 on dist-test-slave-hfc3"
server_key: "f568411950b4aa164147656f7b7b2353"
server_key_iv: "822509907c513478a81c5d2fe9803187"
server_key_version: "encryptionkey@0"
I20260620 14:28:46.232079 15026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:46.241801 15026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:46.243791 15026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:46.243976 15026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:46.244220 15026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:46.244541 15026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:46.244596 15026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:46.244638 15026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:46.244670 15026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:46.250861 15026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40199
I20260620 14:28:46.250928 15154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40199 every 8 connection(s)
I20260620 14:28:46.251313 15026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:28:46.252601 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:46.252167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:58429)
0620 14:28:46.252383 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:46.252457 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
W20260620 14:28:46.253240 15155 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46071 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46071: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
I20260620 14:28:46.253615 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:46.253502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:37283)
0620 14:28:46.253556 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:46.253588 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:28:46.253953 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:46.253871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:51473)
0620 14:28:46.253913 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:46.253937 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:28:46.254086 15155 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:28:46.258026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 15026
I20260620 14:28:47.075794 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:47.075516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:33771)
0620 14:28:47.075684 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:47.075729 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260620 14:28:47.134788 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:47.134574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:42381)
0620 14:28:47.134693 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:47.134739 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260620 14:28:47.254710 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:47.254489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:46521)
0620 14:28:47.254622 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:47.254660 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260620 14:28:47.561322 15162 raft_consensus.cc:482] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:47.561455 15162 raft_consensus.cc:504] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:47.561750 15162 leader_election.cc:288] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199), 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991)
Jun 20 14:28:47 dist-test-slave-hfc3 krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 14:28:47 dist-test-slave-hfc3 krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 14:28:47.580651 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:47.562109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49657 (local address 127.0.98.3:40199)
0620 14:28:47.562274 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:47.562287 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:47.562322 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:28:47.562940 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:47.562944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:47.563010 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:47.563221 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:47.564243 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:47.565676 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:47.567365 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:47.568022 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:47.568147 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:47.576240 (+ 8093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:47.576245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:47.576246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:47.576259 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:47.578783 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:47.579505 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:47.579508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:47.579509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:47.579588 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:47.579797 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:47.579798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:47.579799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:47.580037 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:47.580124 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:47.580374 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 14:28:47.580514 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":56,"threads_started":1}
I20260620 14:28:47.580652 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:47.562085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50549 (local address 127.0.98.2:36991)
0620 14:28:47.562274 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:47.562287 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:47.562322 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:28:47.562964 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:47.562966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:47.563012 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:47.563170 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:47.564243 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:47.565676 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:47.567352 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:47.568032 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:47.568149 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:47.576493 (+ 8344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:47.576498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:47.576498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:47.576511 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:47.578776 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:47.579319 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:47.579323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:47.579324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:47.579426 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:47.579727 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:47.579729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:47.579729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:47.579999 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:47.580094 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:47.580403 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 14:28:47.580537 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260620 14:28:47.580653 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:47.561964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36991 (local address 127.0.98.1:50549)
0620 14:28:47.562026 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:47.562038 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:47.562096 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:47.563229 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:47.563232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:47.563718 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:47.564074 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:47.564082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:47.565824 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:47.565825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:47.567222 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:47.567225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:47.568132 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:47.568133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:47.568294 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:47.574162 (+ 5868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:47.574172 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:47.576357 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:47.578872 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:47.578880 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:47.578882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:47.579227 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:47.579509 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:47.579510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:47.579510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:47.579621 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:47.580074 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:47.580081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:47.580211 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 14:28:47.580321 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:28:47.580531 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":93}
I20260620 14:28:47.580778 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:47.561964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40199 (local address 127.0.98.1:49657)
0620 14:28:47.562081 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:47.562090 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:47.562130 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:47.563229 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:47.563232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:47.563979 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:47.564073 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:47.564082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:47.565811 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:47.565813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:47.567221 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:47.567225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:47.568268 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:47.568269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:47.568325 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:47.574181 (+ 5856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:47.574186 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:47.576040 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:47.578919 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:47.578920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:47.578921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:47.579227 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:47.579659 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:47.579660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:47.579661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:47.579721 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:47.580119 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:47.580120 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:47.580212 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 14:28:47.580320 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:28:47.580613 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":219,"thread_start_us":55,"threads_started":1}
W20260620 14:28:47.581811 14776 leader_election.cc:341] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199): Not found: Tablet not found: ad625f3e60b54ef2beafaf8f38a3666e
W20260620 14:28:47.581911 14775 leader_election.cc:341] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991): Not found: Tablet not found: ad625f3e60b54ef2beafaf8f38a3666e
I20260620 14:28:47.582008 14775 leader_election.cc:302] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59cc386a6f484094b8b40f15b9977a65; no voters: 6115885c5692438c88137fac7d081af1, bda0b3ebe98d4d78915c4eb59126fcd9
I20260620 14:28:47.582124 15162 raft_consensus.cc:2726] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:28:48.076413 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:48.076274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:43667)
0620 14:28:48.076336 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:48.076373 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:28:48.135396 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:48.135209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:57163)
0620 14:28:48.135322 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:48.135359 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260620 14:28:48.255329 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:48.255130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:52577)
0620 14:28:48.255257 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:48.255293 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260620 14:28:49.077211 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:49.076938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:50319)
0620 14:28:49.077094 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:49.077144 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260620 14:28:49.136111 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:49.135862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:57287)
0620 14:28:49.136022 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:49.136069 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260620 14:28:49.255995 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:49.255793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:42387)
0620 14:28:49.255913 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:49.255953 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260620 14:28:49.299194 15171 raft_consensus.cc:482] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:49.299316 15171 raft_consensus.cc:504] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:49.299548 15171 leader_election.cc:288] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199), 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991)
W20260620 14:28:49.299973 14776 leader_election.cc:341] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199): Not found: Tablet not found: ad625f3e60b54ef2beafaf8f38a3666e
W20260620 14:28:49.300094 14775 leader_election.cc:341] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991): Not found: Tablet not found: ad625f3e60b54ef2beafaf8f38a3666e
I20260620 14:28:49.300172 14775 leader_election.cc:302] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59cc386a6f484094b8b40f15b9977a65; no voters: 6115885c5692438c88137fac7d081af1, bda0b3ebe98d4d78915c4eb59126fcd9
I20260620 14:28:49.300280 15171 raft_consensus.cc:2726] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:28:50.077967 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:50.077735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:34153)
0620 14:28:50.077880 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:50.077919 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260620 14:28:50.136751 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:50.136534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:50909)
0620 14:28:50.136667 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:50.136706 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260620 14:28:50.256682 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:50.256451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:37253)
0620 14:28:50.256601 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:50.256640 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260620 14:28:51.078747 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:51.078548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:40769)
0620 14:28:51.078667 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:51.078700 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260620 14:28:51.137404 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:51.137211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:56133)
0620 14:28:51.137330 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:51.137363 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260620 14:28:51.257477 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:51.257236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:37859)
0620 14:28:51.257394 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:51.257436 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260620 14:28:52.058416 15178 raft_consensus.cc:482] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:52.058537 15178 raft_consensus.cc:504] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:52.058737 15178 leader_election.cc:288] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199), 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991)
W20260620 14:28:52.059131 14776 leader_election.cc:341] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199): Not found: Tablet not found: ad625f3e60b54ef2beafaf8f38a3666e
W20260620 14:28:52.059185 14775 leader_election.cc:341] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991): Not found: Tablet not found: ad625f3e60b54ef2beafaf8f38a3666e
I20260620 14:28:52.059214 14776 leader_election.cc:302] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59cc386a6f484094b8b40f15b9977a65; no voters: 6115885c5692438c88137fac7d081af1, bda0b3ebe98d4d78915c4eb59126fcd9
I20260620 14:28:52.059296 15178 raft_consensus.cc:2726] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:28:52.079525 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:52.079250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:52203)
0620 14:28:52.079417 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:52.079476 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
I20260620 14:28:52.138087 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:52.137868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:56437)
0620 14:28:52.138005 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:52.138045 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260620 14:28:52.258373 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:52.258055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:34869)
0620 14:28:52.258251 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:52.258288 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":131,"threads_started":1}
I20260620 14:28:53.080206 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:53.079977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:53831)
0620 14:28:53.080119 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:53.080157 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260620 14:28:53.138777 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:53.138578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:48675)
0620 14:28:53.138702 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:53.138736 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260620 14:28:53.259047 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:53.258821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:41731)
0620 14:28:53.258963 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:53.259004 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260620 14:28:54.080909 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:54.080673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:41547)
0620 14:28:54.080809 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:54.080849 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260620 14:28:54.139479 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:54.139264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:39111)
0620 14:28:54.139392 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:54.139436 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260620 14:28:54.259739 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:54.259538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:50265)
0620 14:28:54.259660 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:54.259694 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260620 14:28:55.081799 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.081531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:33005)
0620 14:28:55.081713 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.081752 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260620 14:28:55.140123 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.139911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:60505)
0620 14:28:55.140041 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.140079 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260620 14:28:55.260470 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.260260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:44009)
0620 14:28:55.260392 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.260429 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260620 14:28:56.082504 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:56.082261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:41907)
0620 14:28:56.082417 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:56.082456 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":60,"threads_started":1}
I20260620 14:28:56.140808 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:56.140581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:59061)
0620 14:28:56.140721 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:56.140765 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260620 14:28:56.261159 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:56.260930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:57935)
0620 14:28:56.261087 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:56.261118 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260620 14:28:56.577993 15194 raft_consensus.cc:482] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:56.578120 15194 raft_consensus.cc:504] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:56.578341 15194 leader_election.cc:288] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199), 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991)
W20260620 14:28:56.578783 14776 leader_election.cc:341] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199): Not found: Tablet not found: ad625f3e60b54ef2beafaf8f38a3666e
W20260620 14:28:56.578866 14775 leader_election.cc:341] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991): Not found: Tablet not found: ad625f3e60b54ef2beafaf8f38a3666e
I20260620 14:28:56.578915 14775 leader_election.cc:302] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59cc386a6f484094b8b40f15b9977a65; no voters: 6115885c5692438c88137fac7d081af1, bda0b3ebe98d4d78915c4eb59126fcd9
I20260620 14:28:56.578989 15194 raft_consensus.cc:2726] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:28:57.083173 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:57.082952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:35339)
0620 14:28:57.083089 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:57.083127 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260620 14:28:57.141481 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:57.141281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:40007)
0620 14:28:57.141400 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:57.141440 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260620 14:28:57.261842 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:57.261608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:57799)
0620 14:28:57.261752 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:57.261787 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260620 14:28:58.083909 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:58.083673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:33225)
0620 14:28:58.083818 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:58.083862 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260620 14:28:58.142208 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:58.141965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:47479)
0620 14:28:58.142111 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:58.142154 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260620 14:28:58.262549 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:58.262284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:60385)
0620 14:28:58.262462 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:58.262503 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":102,"threads_started":1}
I20260620 14:28:59.084719 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:59.084510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:36553)
0620 14:28:59.084628 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:59.084670 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260620 14:28:59.142928 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:59.142711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:41899)
0620 14:28:59.142839 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:59.142883 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260620 14:28:59.263247 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:59.263022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:42323)
0620 14:28:59.263146 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:59.263186 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260620 14:28:59.814975 15204 raft_consensus.cc:482] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:59.815076 15204 raft_consensus.cc:504] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda0b3ebe98d4d78915c4eb59126fcd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40199 } } peers { permanent_uuid: "6115885c5692438c88137fac7d081af1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "59cc386a6f484094b8b40f15b9977a65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44759 } }
I20260620 14:28:59.815281 15204 leader_election.cc:288] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199), 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991)
W20260620 14:28:59.815732 14775 leader_election.cc:341] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6115885c5692438c88137fac7d081af1 (127.0.98.2:36991): Not found: Tablet not found: ad625f3e60b54ef2beafaf8f38a3666e
W20260620 14:28:59.815828 14776 leader_election.cc:341] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bda0b3ebe98d4d78915c4eb59126fcd9 (127.0.98.3:40199): Not found: Tablet not found: ad625f3e60b54ef2beafaf8f38a3666e
I20260620 14:28:59.815877 14776 leader_election.cc:302] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59cc386a6f484094b8b40f15b9977a65; no voters: 6115885c5692438c88137fac7d081af1, bda0b3ebe98d4d78915c4eb59126fcd9
I20260620 14:28:59.815979 15204 raft_consensus.cc:2726] T ad625f3e60b54ef2beafaf8f38a3666e P 59cc386a6f484094b8b40f15b9977a65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:29:00.085495 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:00.085253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:52247)
0620 14:29:00.085386 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:00.085448 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260620 14:29:00.143683 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:00.143391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:50569)
0620 14:29:00.143591 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:00.143638 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":125,"threads_started":1}
I20260620 14:29:00.263945 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:00.263701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:57607)
0620 14:29:00.263852 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:00.263895 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260620 14:29:01.086440 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:01.086167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:44695)
0620 14:29:01.086300 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:01.086395 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":93,"threads_started":1}
I20260620 14:29:01.144523 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:01.144262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:55899)
0620 14:29:01.144394 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:01.144476 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260620 14:29:01.264844 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:01.264594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:57517)
0620 14:29:01.264752 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:01.264798 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260620 14:29:01.269289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 14730
I20260620 14:29:01.274504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 14894
I20260620 14:29:01.279042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 15026
2026-06-20T14:29:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19536 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:01 dist-test-slave-hfc3 krb5kdc[15213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:01 dist-test-slave-hfc3 krb5kdc[15213](info): set up 2 sockets
Jun 20 14:29:01 dist-test-slave-hfc3 krb5kdc[15213](info): commencing operation
krb5kdc: starting...
W20260620 14:29:02.822582 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:02 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:29:02.941190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32893
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:03.023344 15226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:03.023566 15226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:03.023587 15226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:03.024748 15226 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:29:03.024792 15226 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:29:03.024806 15226 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:29:03.024859 15226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:03.024891 15226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:03.024919 15226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:03.024938 15226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:03.024968 15226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:03.026497 15226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:29:03.026751 15226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:03.026993 15226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:03.031975 15231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:03.031982 15232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:03.032013 15234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:03.032066 15226 server_base.cc:1034] running on GCE node
I20260620 14:29:03.032424 15226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:03.032730 15226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:03.033895 15226 hybrid_clock.cc:648] HybridClock initialized: now 1781965743033867 us; error 46 us; skew 500 ppm
Jun 20 14:29:03 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:03.036363 15226 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:29:03.036767 15226 webserver.cc:460] Webserver started at http://127.0.98.62:34985/ using document root <none> and password file <none>
I20260620 14:29:03.036948 15226 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:03.036989 15226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:03.037075 15226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:03.038079 15226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "8296c81c74724ea099cae5a5ca8770cc"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "608f7d57c5bb931be9f9fdf014eea33f"
server_key_iv: "2ac5acae09f2feee294cda6708af5241"
server_key_version: "encryptionkey@0"
I20260620 14:29:03.038457 15226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "8296c81c74724ea099cae5a5ca8770cc"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "608f7d57c5bb931be9f9fdf014eea33f"
server_key_iv: "2ac5acae09f2feee294cda6708af5241"
server_key_version: "encryptionkey@0"
I20260620 14:29:03.039834 15226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:03.040848 15241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:03.041003 15226 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:29:03.041090 15226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "8296c81c74724ea099cae5a5ca8770cc"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "608f7d57c5bb931be9f9fdf014eea33f"
server_key_iv: "2ac5acae09f2feee294cda6708af5241"
server_key_version: "encryptionkey@0"
I20260620 14:29:03.041149 15226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:03.057439 15226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:03.059756 15226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:03.059877 15226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:03.063766 15226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32893
I20260620 14:29:03.063854 15293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32893 every 8 connection(s)
I20260620 14:29:03.064188 15226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:03.065089 15294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:03.065748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 15226
I20260620 14:29:03.065860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:03.066071 392 external_mini_cluster.cc:1428] Setting key 4aa5577def91b931c3d3d7da3ec48915
I20260620 14:29:03.068310 15294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Bootstrap starting.
I20260620 14:29:03.069036 15294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:03.069332 15294 log.cc:824] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:03.070010 15294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: No bootstrap required, opened a new log
I20260620 14:29:03.072211 15294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260620 14:29:03.072350 15294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:03.072372 15294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8296c81c74724ea099cae5a5ca8770cc, State: Initialized, Role: FOLLOWER
I20260620 14:29:03.072505 15294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260620 14:29:03.072607 15294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:03.072654 15294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:03.072705 15294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:29:03 dist-test-slave-hfc3 krb5kdc[15213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:03.073328 15294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260620 14:29:03.073469 15294 leader_election.cc:302] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8296c81c74724ea099cae5a5ca8770cc; no voters:
I20260620 14:29:03.073678 15294 leader_election.cc:288] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:03.073735 15299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:03.073892 15299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 1 LEADER]: Becoming Leader. State: Replica: 8296c81c74724ea099cae5a5ca8770cc, State: Running, Role: LEADER
I20260620 14:29:03.074013 15299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260620 14:29:03.074064 15294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:03.074401 15299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8296c81c74724ea099cae5a5ca8770cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } } }
I20260620 14:29:03.074484 15299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:03.074635 15299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8296c81c74724ea099cae5a5ca8770cc. Latest consensus state: current_term: 1 leader_uuid: "8296c81c74724ea099cae5a5ca8770cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } } }
I20260620 14:29:03.074762 15303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:03.074771 15299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:03.075314 15303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:03.077962 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.066667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.0.98.62:32893)
0620 14:29:03.066879 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:03.066889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:03.066922 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:29:03.067466 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:03.067475 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:03.067539 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:03.067692 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:03.068296 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.069676 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.070479 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.071118 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.071237 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.073784 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:03.073789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:03.073790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:03.073802 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:03.076437 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:03.076810 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:03.076813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:03.076814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:03.076905 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:03.077118 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:03.077119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:03.077119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:03.077358 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:03.077517 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:03.077635 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 14:29:03.077775 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":60,"threads_started":1}
I20260620 14:29:03.079097 15303 catalog_manager.cc:1347] Generated new cluster ID: 997be8ca6dd64a1cbb878514405073f2
I20260620 14:29:03.079150 15303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:03.088332 15303 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:03.089077 15303 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:03.104141 15303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Generated new TSK 0
I20260620 14:29:03.104377 15303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:29:03.133198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32893
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:03.215818 15320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:03.216058 15320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:03.216096 15320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:03.217233 15320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:03.217296 15320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:03.217342 15320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:29:03.218861 15320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:29:03.219121 15320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:03.219339 15320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:03.224189 15325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:03.224327 15320 server_base.cc:1034] running on GCE node
W20260620 14:29:03.224257 15328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:03.224205 15326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:03.224715 15320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:03.224965 15320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:03.226119 15320 hybrid_clock.cc:648] HybridClock initialized: now 1781965743226113 us; error 36 us; skew 500 ppm
Jun 20 14:29:03 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:03.228628 15320 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:29:03.229041 15320 webserver.cc:460] Webserver started at http://127.0.98.1:38795/ using document root <none> and password file <none>
I20260620 14:29:03.229269 15320 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:03.229313 15320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:03.229427 15320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:03.230515 15320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "bae2ad84973e4f45b807a4a19e5b69f0"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "1045301dea044da71ed2b177820ec80b"
server_key_iv: "c2425c2117cb5bed16ece342b5d39b9d"
server_key_version: "encryptionkey@0"
I20260620 14:29:03.230873 15320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bae2ad84973e4f45b807a4a19e5b69f0"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "1045301dea044da71ed2b177820ec80b"
server_key_iv: "c2425c2117cb5bed16ece342b5d39b9d"
server_key_version: "encryptionkey@0"
I20260620 14:29:03.232214 15320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:03.232918 15335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:03.233080 15320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:29:03.233177 15320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "bae2ad84973e4f45b807a4a19e5b69f0"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "1045301dea044da71ed2b177820ec80b"
server_key_iv: "c2425c2117cb5bed16ece342b5d39b9d"
server_key_version: "encryptionkey@0"
I20260620 14:29:03.233256 15320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:03.258708 15320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:03.261215 15320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:03.261384 15320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:03.261632 15320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:03.261958 15320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:03.262017 15320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:03.262065 15320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:03.262095 15320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:03.268216 15320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36811
I20260620 14:29:03.268263 15448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36811 every 8 connection(s)
I20260620 14:29:03.268600 15320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:29:03 dist-test-slave-hfc3 krb5kdc[15213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:03.277907 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.269313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50149 (local address 127.0.98.62:32893)
0620 14:29:03.269378 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:03.269380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:03.269601 (+ 221us) server_negotiation.cc:407] Connection header received
0620 14:29:03.270098 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:03.270099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:03.270109 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:03.270175 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:03.271153 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.271661 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.272702 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.272903 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.272956 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.275390 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:03.275392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:03.275392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:03.275399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:03.276718 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:03.277100 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:03.277101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:03.277101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:03.277163 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:03.277375 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:03.277376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:03.277376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:03.277485 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:03.277534 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:03.277771 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 14:29:03.277839 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:29:03.278131 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.269298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:50149)
0620 14:29:03.269462 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:03.269506 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:03.269987 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:03.270240 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:03.270243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:03.270860 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:03.271067 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.271072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.271788 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.271789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.272613 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.272615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.272979 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.272980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.273087 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.273685 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:03.273694 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:03.275306 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:03.276792 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:03.276798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:03.276800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:03.277035 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:03.277218 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:03.277219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:03.277219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:03.277322 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:03.277536 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:03.277540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:03.277661 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 14:29:03.277745 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:29:03.277869 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260620 14:29:03.278671 15449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32893
I20260620 14:29:03.278774 15449 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:03.278777 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 15320
I20260620 14:29:03.278888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:29:03.278983 15449 heartbeater.cc:511] Master 127.0.98.62:32893 requested a full tablet report, sending...
I20260620 14:29:03.279129 392 external_mini_cluster.cc:1428] Setting key 3a6f1a37c02e678d34f89b5da824e221
I20260620 14:29:03.279541 15258 ts_manager.cc:194] Registered new tserver with Master: bae2ad84973e4f45b807a4a19e5b69f0 (127.0.98.1:36811)
I20260620 14:29:03.280200 15258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50149
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:29:03.305174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32893
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:03.386054 15454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:03.386312 15454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:03.386377 15454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:03.387522 15454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:03.387584 15454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:03.387629 15454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:29:03.389038 15454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:29:03.389292 15454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:03.389516 15454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:03.394376 15462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:03.394330 15460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:03.394359 15459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:03.394481 15454 server_base.cc:1034] running on GCE node
I20260620 14:29:03.394745 15454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:03.394987 15454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:03.396126 15454 hybrid_clock.cc:648] HybridClock initialized: now 1781965743396107 us; error 36 us; skew 500 ppm
Jun 20 14:29:03 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:03.398586 15454 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:29:03.399020 15454 webserver.cc:460] Webserver started at http://127.0.98.2:36567/ using document root <none> and password file <none>
I20260620 14:29:03.399278 15454 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:03.399327 15454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:03.399470 15454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:03.400581 15454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "93b87b7777e74e9c9bcee013eb506ca7"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "251b237e518ad54c60543337fd7b5445"
server_key_iv: "50f361390bfa851bb5edce9aebd5de27"
server_key_version: "encryptionkey@0"
I20260620 14:29:03.400939 15454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "93b87b7777e74e9c9bcee013eb506ca7"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "251b237e518ad54c60543337fd7b5445"
server_key_iv: "50f361390bfa851bb5edce9aebd5de27"
server_key_version: "encryptionkey@0"
I20260620 14:29:03.402244 15454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:29:03.403029 15469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:03.403223 15454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:29:03.403295 15454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "93b87b7777e74e9c9bcee013eb506ca7"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "251b237e518ad54c60543337fd7b5445"
server_key_iv: "50f361390bfa851bb5edce9aebd5de27"
server_key_version: "encryptionkey@0"
I20260620 14:29:03.403376 15454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:03.426841 15454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:03.429018 15454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:03.429178 15454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:03.429425 15454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:03.429746 15454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:03.429781 15454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:03.429804 15454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:03.429848 15454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:03.436036 15454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34879
I20260620 14:29:03.436115 15582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34879 every 8 connection(s)
I20260620 14:29:03.436441 15454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:29:03.440483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 15454
I20260620 14:29:03.440600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:29:03.440826 392 external_mini_cluster.cc:1428] Setting key 0f3109547ba0ff664a7e191dd7517e6f
Jun 20 14:29:03 dist-test-slave-hfc3 krb5kdc[15213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:03.446220 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.437205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49989 (local address 127.0.98.62:32893)
0620 14:29:03.437293 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:03.437296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:03.437530 (+ 234us) server_negotiation.cc:407] Connection header received
0620 14:29:03.438020 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:03.438021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:03.438031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:03.438083 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:03.439050 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.439545 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.440684 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.440908 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.440968 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.443340 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:03.443342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:03.443342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:03.443348 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:03.444816 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:03.445229 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:03.445230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:03.445231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:03.445289 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:03.445568 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:03.445569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:03.445569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:03.445687 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:03.445758 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:03.446036 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:29:03.446117 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:29:03.446475 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.437220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:49989)
0620 14:29:03.437381 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:03.437427 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:03.437934 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:03.438136 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:03.438139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:03.438770 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:03.438960 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.438965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.439621 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.439622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.440547 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.440550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.440997 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.440998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.441145 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.441699 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:03.441707 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:03.443249 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:03.444901 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:03.444907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:03.444909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:03.445156 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:03.445396 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:03.445397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:03.445397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:03.445500 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:03.445750 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:03.445756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:03.445894 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 14:29:03.446003 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:29:03.446159 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260620 14:29:03.446888 15583 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32893
I20260620 14:29:03.446979 15583 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:03.447171 15583 heartbeater.cc:511] Master 127.0.98.62:32893 requested a full tablet report, sending...
I20260620 14:29:03.447526 15258 ts_manager.cc:194] Registered new tserver with Master: 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879)
I20260620 14:29:03.447892 15258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49989
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:29:03.464793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32893
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:03.546320 15588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:03.546567 15588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:03.546604 15588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:03.547717 15588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:03.547777 15588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:03.547819 15588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:29:03.549247 15588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:29:03.549508 15588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:03.549736 15588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:03.554629 15596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:03.554644 15593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:03.554785 15594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:03.554942 15588 server_base.cc:1034] running on GCE node
I20260620 14:29:03.555127 15588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:03.555341 15588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:03.556480 15588 hybrid_clock.cc:648] HybridClock initialized: now 1781965743556469 us; error 28 us; skew 500 ppm
Jun 20 14:29:03 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:03.559232 15588 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:29:03.559648 15588 webserver.cc:460] Webserver started at http://127.0.98.3:38349/ using document root <none> and password file <none>
I20260620 14:29:03.559886 15588 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:03.559931 15588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:03.560089 15588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:03.561216 15588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "0452373f42024dbcb07d1840c5075147"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "fb7a42fcfad8a32ecceb62940de284c8"
server_key_iv: "721bd0bcaef9e3763131a933e346eb81"
server_key_version: "encryptionkey@0"
I20260620 14:29:03.561563 15588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0452373f42024dbcb07d1840c5075147"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "fb7a42fcfad8a32ecceb62940de284c8"
server_key_iv: "721bd0bcaef9e3763131a933e346eb81"
server_key_version: "encryptionkey@0"
I20260620 14:29:03.562933 15588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:29:03.563654 15603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:03.563850 15588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:29:03.563939 15588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "0452373f42024dbcb07d1840c5075147"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "fb7a42fcfad8a32ecceb62940de284c8"
server_key_iv: "721bd0bcaef9e3763131a933e346eb81"
server_key_version: "encryptionkey@0"
I20260620 14:29:03.564009 15588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:03.584865 15588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:03.586910 15588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:03.587065 15588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:03.587301 15588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:03.587618 15588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:03.587652 15588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:03.587677 15588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:03.587730 15588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:03.593717 15588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36145
I20260620 14:29:03.593811 15716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36145 every 8 connection(s)
I20260620 14:29:03.594117 15588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:29:03.599591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 15588
I20260620 14:29:03.599670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:29:03.599862 392 external_mini_cluster.cc:1428] Setting key d15068d6d0f28904e6c148be27c8aee2
Jun 20 14:29:03 dist-test-slave-hfc3 krb5kdc[15213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:03.603194 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.594856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53571 (local address 127.0.98.62:32893)
0620 14:29:03.594923 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:03.594925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:03.595109 (+ 184us) server_negotiation.cc:407] Connection header received
0620 14:29:03.595590 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:03.595591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:03.595601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:03.595655 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:03.596587 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.597087 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.598054 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.598229 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.598278 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.600648 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:03.600650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:03.600650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:03.600658 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:03.601820 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:03.602214 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:03.602215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:03.602215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:03.602278 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:03.602546 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:03.602547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:03.602548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:03.602667 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:03.602746 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:03.603022 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 14:29:03.603104 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:29:03.603384 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.594847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:53571)
0620 14:29:03.594997 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:03.595034 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:03.595505 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:03.595730 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:03.595733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:03.596311 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:03.596508 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.596512 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.597163 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.597163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.597982 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.597985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.598296 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.598297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.598474 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.598929 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:03.598937 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:03.600552 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:03.601900 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:03.601907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:03.601910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:03.602132 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:03.602360 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:03.602361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:03.602361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:03.602479 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:03.602736 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:03.602742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:03.602886 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 14:29:03.602986 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:29:03.603143 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260620 14:29:03.603864 15717 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32893
I20260620 14:29:03.603971 15717 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:03.604157 15717 heartbeater.cc:511] Master 127.0.98.62:32893 requested a full tablet report, sending...
I20260620 14:29:03.604540 15258 ts_manager.cc:194] Registered new tserver with Master: 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145)
I20260620 14:29:03.604946 15258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53571
I20260620 14:29:03.611768 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:29:03.619999 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.613456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.0.98.62:32893)
0620 14:29:03.613492 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:03.613494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:03.613646 (+ 152us) server_negotiation.cc:407] Connection header received
0620 14:29:03.613734 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:03.613735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:03.613744 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:03.613789 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:03.614358 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.614843 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.615898 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.616179 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.616252 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.617809 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:03.617810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:03.617811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:03.617819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:03.619070 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:03.619378 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:03.619379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:03.619379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:03.619428 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:03.619568 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:03.619569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:03.619569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:03.619707 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:03.619777 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:03.619859 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:29:03.619917 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:29:03.621404 15258 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:29:03.623144 15258 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:29:03.629431 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.625413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36145 (local address 127.0.0.1:51320)
0620 14:29:03.625758 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:03.625767 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:03.625806 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:03.626013 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:03.626016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:03.626017 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:03.626069 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.626070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.627108 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.627110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.628180 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.628182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.629138 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.629139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.629224 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.629238 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:29:03.629307 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:29:03.629355 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":102,"threads_started":1}
I20260620 14:29:03.629674 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.625650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.0.98.3:36145)
0620 14:29:03.625841 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:03.625847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:03.625858 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:29:03.625901 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:03.625903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:03.625915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:03.625996 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:03.626146 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.627008 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.628285 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.628964 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.629046 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.629105 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:03.629550 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 14:29:03.629599 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260620 14:29:03.629704 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.625401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36811 (local address 127.0.0.1:56278)
0620 14:29:03.625566 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:03.625609 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:03.625678 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:03.626253 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:03.626254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:03.626256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:03.626311 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.626313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.627660 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.627661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.628735 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.628737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.629467 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.629468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.629534 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.629538 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:29:03.629583 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:29:03.629622 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260620 14:29:03.629799 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.625582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56278 (local address 127.0.98.1:36811)
0620 14:29:03.626107 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:03.626111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:03.626119 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:29:03.626149 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:03.626150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:03.626160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:03.626254 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:03.626438 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.627552 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.628827 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.629361 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.629467 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.629636 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:03.629663 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 14:29:03.629706 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":88,"threads_started":1}
I20260620 14:29:03.630380 15651 tablet_service.cc:1467] Processing CreateTablet for tablet e9a5877759df410195fd10237e596e52 (DEFAULT_TABLE table=test-table [id=69cb9bce5e034f42a4d1023d357aea3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:03.630538 15383 tablet_service.cc:1467] Processing CreateTablet for tablet e9a5877759df410195fd10237e596e52 (DEFAULT_TABLE table=test-table [id=69cb9bce5e034f42a4d1023d357aea3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:03.630738 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.625413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34879 (local address 127.0.0.1:51004)
0620 14:29:03.626380 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:03.626390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:03.626439 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:03.626712 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:03.626713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:03.626715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:03.626760 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.626761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.627915 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.627916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.628937 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.628940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.630503 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.630504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.630566 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.630571 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:29:03.630625 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:29:03.630665 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":12,"thread_start_us":85,"threads_started":1}
I20260620 14:29:03.630803 15651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a5877759df410195fd10237e596e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:03.630821 15383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a5877759df410195fd10237e596e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:03.630965 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.625663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.0.98.2:34879)
0620 14:29:03.625841 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:03.625847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:03.626503 (+ 656us) server_negotiation.cc:407] Connection header received
0620 14:29:03.626549 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:03.626552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:03.626566 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:03.626633 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:03.626836 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.627816 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.629026 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.630295 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.630499 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.630561 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:03.630843 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 14:29:03.630884 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260620 14:29:03.634117 15735 tablet_bootstrap.cc:492] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147: Bootstrap starting.
I20260620 14:29:03.634545 15734 tablet_bootstrap.cc:492] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Bootstrap starting.
I20260620 14:29:03.634785 15517 tablet_service.cc:1467] Processing CreateTablet for tablet e9a5877759df410195fd10237e596e52 (DEFAULT_TABLE table=test-table [id=69cb9bce5e034f42a4d1023d357aea3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:03.634966 15735 tablet_bootstrap.cc:654] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:03.635105 15517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a5877759df410195fd10237e596e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:03.635299 15734 tablet_bootstrap.cc:654] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:03.635617 15735 log.cc:824] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:03.635891 15734 log.cc:824] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:03.636484 15735 tablet_bootstrap.cc:492] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147: No bootstrap required, opened a new log
I20260620 14:29:03.636562 15735 ts_tablet_manager.cc:1397] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:29:03.636673 15734 tablet_bootstrap.cc:492] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: No bootstrap required, opened a new log
I20260620 14:29:03.636737 15734 ts_tablet_manager.cc:1397] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:03.638183 15738 tablet_bootstrap.cc:492] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7: Bootstrap starting.
I20260620 14:29:03.638818 15738 tablet_bootstrap.cc:654] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:03.638720 15735 raft_consensus.cc:348] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } }
I20260620 14:29:03.638708 15734 raft_consensus.cc:348] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } }
I20260620 14:29:03.638888 15735 raft_consensus.cc:374] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:03.638888 15734 raft_consensus.cc:374] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:03.638934 15734 raft_consensus.cc:729] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae2ad84973e4f45b807a4a19e5b69f0, State: Initialized, Role: FOLLOWER
I20260620 14:29:03.638934 15735 raft_consensus.cc:729] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0452373f42024dbcb07d1840c5075147, State: Initialized, Role: FOLLOWER
I20260620 14:29:03.639047 15734 consensus_queue.cc:260] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } }
I20260620 14:29:03.639065 15735 consensus_queue.cc:260] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } }
I20260620 14:29:03.639343 15735 ts_tablet_manager.cc:1428] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:29:03.639343 15734 ts_tablet_manager.cc:1428] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:29:03.639384 15449 heartbeater.cc:503] Master 127.0.98.62:32893 was elected leader, sending a full tablet report...
I20260620 14:29:03.639434 15738 log.cc:824] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:03.639453 15717 heartbeater.cc:503] Master 127.0.98.62:32893 was elected leader, sending a full tablet report...
I20260620 14:29:03.640156 15738 tablet_bootstrap.cc:492] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7: No bootstrap required, opened a new log
I20260620 14:29:03.640230 15738 ts_tablet_manager.cc:1397] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:03.641662 15738 raft_consensus.cc:348] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } }
I20260620 14:29:03.641824 15738 raft_consensus.cc:374] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:03.641872 15738 raft_consensus.cc:729] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b87b7777e74e9c9bcee013eb506ca7, State: Initialized, Role: FOLLOWER
I20260620 14:29:03.641989 15738 consensus_queue.cc:260] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } }
I20260620 14:29:03.642251 15738 ts_tablet_manager.cc:1428] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:03.642264 15583 heartbeater.cc:503] Master 127.0.98.62:32893 was elected leader, sending a full tablet report...
W20260620 14:29:03.687059 15584 tablet.cc:2369] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:29:03.706766 15739 raft_consensus.cc:482] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:03.706925 15739 raft_consensus.cc:504] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } }
I20260620 14:29:03.707345 15739 leader_election.cc:288] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879), 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145)
I20260620 14:29:03.710171 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.707623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56385 (local address 127.0.98.2:34879)
0620 14:29:03.707669 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:03.707672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:03.707680 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:29:03.707719 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:03.707720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:03.707733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:03.707792 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:03.707975 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.708490 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.709432 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.709813 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.709867 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.709876 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:03.710052 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:29:03.710096 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:29:03.710182 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.707489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34879 (local address 127.0.98.1:56385)
0620 14:29:03.707540 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:03.707554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:03.707607 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:03.707815 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:03.707816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:03.707818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:03.707901 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.707903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.708572 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.708573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.709339 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.709341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.709892 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.709893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.709975 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.709979 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:29:03.710027 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:29:03.710078 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:29:03.710193 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.707610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45955 (local address 127.0.98.3:36145)
0620 14:29:03.707665 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:03.707667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:03.707793 (+ 126us) server_negotiation.cc:407] Connection header received
0620 14:29:03.707822 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:03.707823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:03.707837 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:03.707904 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:03.708074 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.708550 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.709471 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.709835 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.709921 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.709931 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:03.710090 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:29:03.710132 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:29:03.710217 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.707532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36145 (local address 127.0.98.1:45955)
0620 14:29:03.707716 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:03.707725 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:03.707779 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:03.707915 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:03.707916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:03.707917 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:03.708005 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.708006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.708636 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.708637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.709357 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:03.709358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.709936 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.709937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:03.709999 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.710002 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:29:03.710073 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:29:03.710107 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":124,"threads_started":1}
I20260620 14:29:03.710547 15537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a5877759df410195fd10237e596e52" candidate_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b87b7777e74e9c9bcee013eb506ca7" is_pre_election: true
I20260620 14:29:03.710608 15671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a5877759df410195fd10237e596e52" candidate_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0452373f42024dbcb07d1840c5075147" is_pre_election: true
I20260620 14:29:03.710700 15537 raft_consensus.cc:2454] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae2ad84973e4f45b807a4a19e5b69f0 in term 0.
I20260620 14:29:03.710773 15671 raft_consensus.cc:2454] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae2ad84973e4f45b807a4a19e5b69f0 in term 0.
I20260620 14:29:03.710932 15337 leader_election.cc:302] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93b87b7777e74e9c9bcee013eb506ca7, bae2ad84973e4f45b807a4a19e5b69f0; no voters:
I20260620 14:29:03.711097 15739 raft_consensus.cc:2781] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:29:03.711169 15739 raft_consensus.cc:482] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:29:03.711206 15739 raft_consensus.cc:3037] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:03.711910 15739 raft_consensus.cc:504] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } }
I20260620 14:29:03.712051 15739 leader_election.cc:288] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 1 election: Requested vote from peers 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879), 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145)
I20260620 14:29:03.712203 15671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a5877759df410195fd10237e596e52" candidate_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0452373f42024dbcb07d1840c5075147"
I20260620 14:29:03.712217 15537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a5877759df410195fd10237e596e52" candidate_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b87b7777e74e9c9bcee013eb506ca7"
I20260620 14:29:03.712273 15671 raft_consensus.cc:3037] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:03.712302 15537 raft_consensus.cc:3037] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:03.713354 15671 raft_consensus.cc:2454] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae2ad84973e4f45b807a4a19e5b69f0 in term 1.
I20260620 14:29:03.713528 15339 leader_election.cc:302] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0452373f42024dbcb07d1840c5075147, bae2ad84973e4f45b807a4a19e5b69f0; no voters:
I20260620 14:29:03.713646 15739 raft_consensus.cc:2781] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:03.713833 15739 raft_consensus.cc:686] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 LEADER]: Becoming Leader. State: Replica: bae2ad84973e4f45b807a4a19e5b69f0, State: Running, Role: LEADER
I20260620 14:29:03.713930 15739 consensus_queue.cc:237] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } }
I20260620 14:29:03.714330 15537 raft_consensus.cc:2454] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae2ad84973e4f45b807a4a19e5b69f0 in term 1.
I20260620 14:29:03.714754 15258 catalog_manager.cc:5495] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to bae2ad84973e4f45b807a4a19e5b69f0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } health_report { overall_health: UNKNOWN } } }
I20260620 14:29:03.724512 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.722155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.0.98.1:36811)
0620 14:29:03.722197 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:03.722200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:03.722209 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:29:03.722273 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:03.722274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:03.722297 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:03.722407 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:29:03.722519 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.722959 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.723752 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:03.723937 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:03.724017 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:03.724189 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:29:03.724290 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:29:03.724374 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:03.724413 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 14:29:03.724434 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:29:03.726104 15537 raft_consensus.cc:1261] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7 [term 1 FOLLOWER]: Refusing update from remote peer bae2ad84973e4f45b807a4a19e5b69f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:03.726125 15671 raft_consensus.cc:1261] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147 [term 1 FOLLOWER]: Refusing update from remote peer bae2ad84973e4f45b807a4a19e5b69f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:03.726468 15739 consensus_queue.cc:1035] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:03.726600 15744 consensus_queue.cc:1035] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:03.729363 15749 mvcc.cc:205] Tried to move back new op lower bound from 7298931686299377664 to 7298931686252359680. Current Snapshot: MvccSnapshot[applied={T|T < 7298931686299377664}]
I20260620 14:29:03.729449 15748 mvcc.cc:205] Tried to move back new op lower bound from 7298931686299377664 to 7298931686252359680. Current Snapshot: MvccSnapshot[applied={T|T < 7298931686299377664}]
I20260620 14:29:03.731071 15258 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:03.731173 15258 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:03.731884 15258 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Sending DeleteTablet for 3 replicas of tablet e9a5877759df410195fd10237e596e52
I20260620 14:29:03.732173 15383 tablet_service.cc:1514] Processing DeleteTablet for tablet e9a5877759df410195fd10237e596e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56278
I20260620 14:29:03.732223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 15320
I20260620 14:29:03.732205 15651 tablet_service.cc:1514] Processing DeleteTablet for tablet e9a5877759df410195fd10237e596e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51320
I20260620 14:29:03.734223 15517 tablet_service.cc:1514] Processing DeleteTablet for tablet e9a5877759df410195fd10237e596e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51004
I20260620 14:29:03.734452 15757 tablet_replica.cc:330] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7: stopping tablet replica
I20260620 14:29:03.734562 15757 raft_consensus.cc:2229] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:03.734666 15757 raft_consensus.cc:2258] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:03.735062 15757 ts_tablet_manager.cc:1905] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:03.737037 15757 ts_tablet_manager.cc:1918] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:03.737118 15757 log.cc:1196] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/e9a5877759df410195fd10237e596e52
I20260620 14:29:03.737411 15757 ts_tablet_manager.cc:1939] T e9a5877759df410195fd10237e596e52 P 93b87b7777e74e9c9bcee013eb506ca7: Deleting consensus metadata
I20260620 14:29:03.737825 15243 catalog_manager.cc:4841] TS 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879): tablet e9a5877759df410195fd10237e596e52 (table test-table [id=69cb9bce5e034f42a4d1023d357aea3b]) successfully deleted
W20260620 14:29:03.738579 15242 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36811 (error 108)
I20260620 14:29:03.739106 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:03.738969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36811 (local address 127.0.0.1:56302)
0620 14:29:03.739029 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:03.739070 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:29:03.739145 15756 tablet_replica.cc:330] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147: stopping tablet replica
I20260620 14:29:03.739159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 15454
I20260620 14:29:03.739238 15756 raft_consensus.cc:2229] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:03.739338 15756 raft_consensus.cc:2258] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:03.740604 15756 ts_tablet_manager.cc:1905] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:29:03.741016 15242 catalog_manager.cc:4584] TS bae2ad84973e4f45b807a4a19e5b69f0 (127.0.98.1:36811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9a5877759df410195fd10237e596e52: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36811: connect: Connection refused (error 111)
I20260620 14:29:03.742169 15756 ts_tablet_manager.cc:1918] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:03.742242 15756 log.cc:1196] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/e9a5877759df410195fd10237e596e52
I20260620 14:29:03.742537 15756 ts_tablet_manager.cc:1939] T e9a5877759df410195fd10237e596e52 P 0452373f42024dbcb07d1840c5075147: Deleting consensus metadata
I20260620 14:29:03.742923 15245 catalog_manager.cc:4841] TS 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145): tablet e9a5877759df410195fd10237e596e52 (table test-table [id=69cb9bce5e034f42a4d1023d357aea3b]) successfully deleted
I20260620 14:29:03.744479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 15588
I20260620 14:29:03.749428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 15226
I20260620 14:29:23.754866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32893
--webserver_interface=127.0.98.62
--webserver_port=34985
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:23.836724 15759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:23.836949 15759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:23.836971 15759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:23.838171 15759 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:29:23.838220 15759 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:29:23.838234 15759 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:29:23.838245 15759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:23.838258 15759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:23.838302 15759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:23.838361 15759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:23.838379 15759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:23.840042 15759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:29:23.840317 15759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:23.840565 15759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:23.849788 15765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:23.849831 15768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:23.849961 15759 server_base.cc:1034] running on GCE node
W20260620 14:29:23.849788 15766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:23.850292 15759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:23.850690 15759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:23.851878 15759 hybrid_clock.cc:648] HybridClock initialized: now 1781965763851858 us; error 41 us; skew 500 ppm
Jun 20 14:29:23 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:23.854658 15759 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:29:23.855165 15759 webserver.cc:460] Webserver started at http://127.0.98.62:34985/ using document root <none> and password file <none>
I20260620 14:29:23.855425 15759 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:23.855508 15759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:23.856940 15759 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:29:23.857654 15775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:23.857836 15759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:29:23.857963 15759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "8296c81c74724ea099cae5a5ca8770cc"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "608f7d57c5bb931be9f9fdf014eea33f"
server_key_iv: "2ac5acae09f2feee294cda6708af5241"
server_key_version: "encryptionkey@0"
I20260620 14:29:23.858487 15759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:23.885350 15759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:23.887524 15759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:23.887701 15759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:23.892140 15759 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32893
I20260620 14:29:23.892192 15827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32893 every 8 connection(s)
I20260620 14:29:23.892639 15759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:23.894165 15828 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:29:23.894860 15828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Bootstrap starting.
I20260620 14:29:23.900722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 15759
I20260620 14:29:23.901055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36811
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32893
--webserver_port=38795
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:29:23.903836 15828 log.cc:824] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:23.905900 15828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:29:23.906219 15828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Bootstrap complete.
I20260620 14:29:23.908735 15828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260620 14:29:23.908973 15828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8296c81c74724ea099cae5a5ca8770cc, State: Initialized, Role: FOLLOWER
I20260620 14:29:23.909135 15828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260620 14:29:23.909209 15828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:23.909273 15828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:23.909348 15828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:29:23.910660 15828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260620 14:29:23.910773 15828 leader_election.cc:302] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8296c81c74724ea099cae5a5ca8770cc; no voters:
I20260620 14:29:23.910993 15828 leader_election.cc:288] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:29:23.911038 15832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:29:23.911170 15832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 2 LEADER]: Becoming Leader. State: Replica: 8296c81c74724ea099cae5a5ca8770cc, State: Running, Role: LEADER
I20260620 14:29:23.911278 15828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:23.911257 15832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260620 14:29:23.911545 15832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8296c81c74724ea099cae5a5ca8770cc. Latest consensus state: current_term: 2 leader_uuid: "8296c81c74724ea099cae5a5ca8770cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } } }
I20260620 14:29:23.911610 15832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:23.911808 15836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:23.911819 15833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8296c81c74724ea099cae5a5ca8770cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } } }
I20260620 14:29:23.912060 15833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:23.912801 15836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:23.912910 15836 catalog_manager.cc:1259] Loaded cluster ID: 997be8ca6dd64a1cbb878514405073f2
I20260620 14:29:23.912949 15836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:23.914167 15836 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:23.914358 15836 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Loaded TSK: 0
I20260620 14:29:23.926405 15836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Generated new TSK 1
I20260620 14:29:23.926522 15836 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:29:23.986651 15830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:23.986884 15830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:23.986922 15830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:23.988080 15830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:23.988144 15830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:23.988189 15830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:29:23.989692 15830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38795
--tserver_master_addrs=127.0.98.62:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:29:23.989954 15830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:23.990198 15830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:23.996251 15855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:23.996264 15856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:23.996263 15858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:23.996742 15830 server_base.cc:1034] running on GCE node
I20260620 14:29:23.996971 15830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:23.997223 15830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:23.998399 15830 hybrid_clock.cc:648] HybridClock initialized: now 1781965763998383 us; error 31 us; skew 500 ppm
Jun 20 14:29:23 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:24.001080 15830 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:29:24.001637 15830 webserver.cc:460] Webserver started at http://127.0.98.1:38795/ using document root <none> and password file <none>
I20260620 14:29:24.001899 15830 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:24.001960 15830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:24.003486 15830 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:24.004251 15865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:24.004443 15830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:24.004541 15830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "bae2ad84973e4f45b807a4a19e5b69f0"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "1045301dea044da71ed2b177820ec80b"
server_key_iv: "c2425c2117cb5bed16ece342b5d39b9d"
server_key_version: "encryptionkey@0"
I20260620 14:29:24.004905 15830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:24.031769 15830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:24.034204 15830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:24.034425 15830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:24.034691 15830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:24.035156 15872 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:29:24.037036 15830 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:29:24.037110 15830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:29:24.037151 15830 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:29:24.037848 15830 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:29:24.037911 15830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:29:24.037958 15872 tablet_bootstrap.cc:492] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Bootstrap starting.
I20260620 14:29:24.045893 15830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36811
I20260620 14:29:24.046406 15830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:29:24.046902 15979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36811 every 8 connection(s)
I20260620 14:29:24.047186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 15830
I20260620 14:29:24.047521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34879
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32893
--webserver_port=36567
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:29:24.049032 15872 log.cc:824] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:24.051934 15872 tablet_bootstrap.cc:492] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:29:24.052275 15872 tablet_bootstrap.cc:492] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Bootstrap complete.
I20260620 14:29:24.053578 15872 ts_tablet_manager.cc:1397] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
Jun 20 14:29:24 dist-test-slave-hfc3 krb5kdc[15213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:24.055320 15872 raft_consensus.cc:348] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } }
I20260620 14:29:24.055722 15872 raft_consensus.cc:729] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae2ad84973e4f45b807a4a19e5b69f0, State: Initialized, Role: FOLLOWER
I20260620 14:29:24.055912 15872 consensus_queue.cc:260] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } }
I20260620 14:29:24.056253 15872 ts_tablet_manager.cc:1428] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20260620 14:29:24.059092 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.047197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:52785)
0620 14:29:24.047414 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:24.047467 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:24.048480 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:24.049118 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:24.049123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:24.049450 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:24.049692 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.049700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.051126 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.051128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.052134 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.052138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.052841 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.052842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.052976 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.053588 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:24.053596 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:24.055168 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:24.057352 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:24.057359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:24.057361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:24.057600 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:24.057887 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:24.057888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:24.057888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:24.058003 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:24.058444 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:24.058451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:24.058594 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 14:29:24.058708 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 14:29:24.058897 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":124,"threads_started":1}
I20260620 14:29:24.059092 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.047329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52785 (local address 127.0.98.62:32893)
0620 14:29:24.047554 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:24.047567 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:24.047608 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:29:24.048739 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:24.048746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:24.048808 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:24.048968 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:24.049835 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.050998 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.052239 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.052728 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.052852 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.055291 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:24.055295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:24.055296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:24.055308 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:24.057249 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:24.057705 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:24.057708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:24.057708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:24.057812 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:24.058076 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:24.058077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:24.058077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:24.058312 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:24.058463 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:24.058763 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 14:29:24.058895 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260620 14:29:24.059803 15980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32893
I20260620 14:29:24.059913 15980 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:24.060091 15980 heartbeater.cc:511] Master 127.0.98.62:32893 requested a full tablet report, sending...
I20260620 14:29:24.060786 15792 ts_manager.cc:194] Registered new tserver with Master: bae2ad84973e4f45b807a4a19e5b69f0 (127.0.98.1:36811)
I20260620 14:29:24.061896 15792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52785
I20260620 14:29:24.062942 15980 heartbeater.cc:503] Master 127.0.98.62:32893 was elected leader, sending a full tablet report...
Jun 20 14:29:24 dist-test-slave-hfc3 krb5kdc[15213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:29:24.070922 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.061396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36811 (local address 127.0.0.1:39208)
0620 14:29:24.061744 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:24.061777 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:24.061838 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:24.062274 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:24.062277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:24.062650 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:24.062708 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.062711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.064076 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.064076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.064703 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.064704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.065302 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.065304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.065397 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.065904 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:24.065909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:24.067465 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:24.069650 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:24.069658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:24.069660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:24.069897 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:24.070150 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:24.070151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:24.070152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:24.070219 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:24.070592 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:24.070599 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:24.070649 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 14:29:24.070732 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:29:24.070849 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":93,"threads_started":1}
I20260620 14:29:24.070989 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.061596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.0.98.1:36811)
0620 14:29:24.061949 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:24.061954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:24.061962 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:29:24.062034 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:24.062036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:24.062050 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:24.062108 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:24.062793 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.063983 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.064782 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.065228 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.065301 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.067558 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:24.067566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:24.067567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:24.067574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:24.069550 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:24.069994 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:24.069998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:24.069998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:24.070065 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:24.070284 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:24.070285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:24.070286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:24.070507 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:24.070591 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:24.070788 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 14:29:24.070889 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":66,"threads_started":1}
I20260620 14:29:24.071602 15914 tablet_service.cc:1514] Processing DeleteTablet for tablet e9a5877759df410195fd10237e596e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39208
I20260620 14:29:24.072316 15989 tablet_replica.cc:330] stopping tablet replica
I20260620 14:29:24.072450 15989 raft_consensus.cc:2229] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:24.072528 15989 raft_consensus.cc:2258] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:24.072822 15989 ts_tablet_manager.cc:1905] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:24.074980 15989 ts_tablet_manager.cc:1918] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:24.075048 15989 log.cc:1196] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal/wals/e9a5877759df410195fd10237e596e52
I20260620 14:29:24.075289 15989 ts_tablet_manager.cc:1939] T e9a5877759df410195fd10237e596e52 P bae2ad84973e4f45b807a4a19e5b69f0: Deleting consensus metadata
I20260620 14:29:24.075824 15776 catalog_manager.cc:4841] TS bae2ad84973e4f45b807a4a19e5b69f0 (127.0.98.1:36811): tablet e9a5877759df410195fd10237e596e52 (table test-table [id=69cb9bce5e034f42a4d1023d357aea3b]) successfully deleted
W20260620 14:29:24.137378 15984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:24.137590 15984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:24.137614 15984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:24.138765 15984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:24.138805 15984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:24.138834 15984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:29:24.140980 15984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36567
--tserver_master_addrs=127.0.98.62:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:29:24.141259 15984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:24.141541 15984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:24.147754 15995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:24.147970 15984 server_base.cc:1034] running on GCE node
W20260620 14:29:24.148065 15997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:24.147754 15994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:24.148351 15984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:24.148658 15984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:24.149814 15984 hybrid_clock.cc:648] HybridClock initialized: now 1781965764149790 us; error 40 us; skew 500 ppm
Jun 20 14:29:24 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:24.152297 15984 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:29:24.152770 15984 webserver.cc:460] Webserver started at http://127.0.98.2:36567/ using document root <none> and password file <none>
I20260620 14:29:24.153018 15984 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:24.153110 15984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:24.154580 15984 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:29:24.155153 16004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:24.155320 15984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:29:24.155422 15984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "93b87b7777e74e9c9bcee013eb506ca7"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "251b237e518ad54c60543337fd7b5445"
server_key_iv: "50f361390bfa851bb5edce9aebd5de27"
server_key_version: "encryptionkey@0"
I20260620 14:29:24.155766 15984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:24.172158 15984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:24.174089 15984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:24.174257 15984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:24.174551 15984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:24.174873 15984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:24.174928 15984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:24.174970 15984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:24.174999 15984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:24.180866 15984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34879
I20260620 14:29:24.180907 16117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34879 every 8 connection(s)
I20260620 14:29:24.181260 15984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:29:24.184029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 15984
I20260620 14:29:24.184324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36145
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32893
--webserver_port=38349
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:29:24 dist-test-slave-hfc3 krb5kdc[15213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:24.191332 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.182007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46273 (local address 127.0.98.62:32893)
0620 14:29:24.182112 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:24.182115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:24.182444 (+ 329us) server_negotiation.cc:407] Connection header received
0620 14:29:24.182953 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:24.182954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:24.182969 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:24.183035 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:24.183969 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.184673 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.185706 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.185934 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.186022 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.188200 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:24.188202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:24.188202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:24.188210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:24.189820 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:24.190266 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:24.190267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:24.190267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:24.190334 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:24.190620 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:24.190621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:24.190621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:24.190749 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:24.190826 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:24.191148 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 14:29:24.191241 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:29:24.191532 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.182062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:46273)
0620 14:29:24.182254 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:24.182291 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:24.182863 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:24.183091 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:24.183094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:24.183660 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:24.183843 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.183857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.184769 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.184771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.185597 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.185600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.186034 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.186035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.186160 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.186735 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:24.186744 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:24.188097 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:24.189935 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:24.189943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:24.189945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:24.190178 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:24.190431 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:24.190432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:24.190433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:24.190545 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:24.190819 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:24.190825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:24.190974 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 14:29:24.191116 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 14:29:24.191278 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
I20260620 14:29:24.192019 16118 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32893
I20260620 14:29:24.192116 16118 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:24.192322 16118 heartbeater.cc:511] Master 127.0.98.62:32893 requested a full tablet report, sending...
I20260620 14:29:24.192741 15792 ts_manager.cc:194] Registered new tserver with Master: 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879)
I20260620 14:29:24.193343 15792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46273
W20260620 14:29:24.268316 16121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:24.268540 16121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:24.268565 16121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:24.269783 16121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:24.269830 16121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:24.269858 16121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:29:24.271420 16121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38349
--tserver_master_addrs=127.0.98.62:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:29:24.271643 16121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:24.271831 16121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:24.277021 16126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:24.277045 16129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:24.277037 16127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:24.277539 16121 server_base.cc:1034] running on GCE node
I20260620 14:29:24.277753 16121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:24.278052 16121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:24.279249 16121 hybrid_clock.cc:648] HybridClock initialized: now 1781965764279250 us; error 48 us; skew 500 ppm
Jun 20 14:29:24 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:24.281853 16121 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:29:24.282287 16121 webserver.cc:460] Webserver started at http://127.0.98.3:38349/ using document root <none> and password file <none>
I20260620 14:29:24.282591 16121 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:24.282657 16121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:24.284173 16121 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:24.284845 16136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:24.285051 16121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:24.285166 16121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "0452373f42024dbcb07d1840c5075147"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "fb7a42fcfad8a32ecceb62940de284c8"
server_key_iv: "721bd0bcaef9e3763131a933e346eb81"
server_key_version: "encryptionkey@0"
I20260620 14:29:24.285585 16121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:24.304941 16121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:24.307508 16121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:24.307686 16121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:24.307934 16121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:24.308351 16121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:24.308413 16121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:24.308460 16121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:24.308497 16121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:24.315881 16121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36145
I20260620 14:29:24.315984 16249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36145 every 8 connection(s)
I20260620 14:29:24.316347 16121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:29:24.320395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 16121
Jun 20 14:29:24 dist-test-slave-hfc3 krb5kdc[15213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:24.326757 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.317217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38243 (local address 127.0.98.62:32893)
0620 14:29:24.317298 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:24.317301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:24.317574 (+ 273us) server_negotiation.cc:407] Connection header received
0620 14:29:24.318046 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:24.318047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:24.318058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:24.318118 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:24.319151 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.319718 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.321023 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.321228 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.321315 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.323542 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:24.323544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:24.323544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:24.323552 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:24.325050 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:24.325458 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:24.325459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:24.325459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:24.325540 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:24.325840 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:24.325841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:24.325841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:24.326065 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:24.326153 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:24.326518 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 14:29:24.326628 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:29:24.326831 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.317226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:38243)
0620 14:29:24.317429 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:24.317466 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:24.317940 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:24.318172 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:24.318175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:24.318845 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:24.319046 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.319051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.319810 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.319811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.320938 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.320941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.321324 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.321325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.321467 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.322027 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:24.322035 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:24.323442 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:24.325149 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:24.325155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:24.325157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:24.325380 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:24.325613 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:24.325614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:24.325614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:24.325765 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:24.326158 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:24.326163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:24.326284 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 14:29:24.326445 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 14:29:24.326629 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260620 14:29:24.327411 16250 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32893
I20260620 14:29:24.327533 16250 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:24.327739 16250 heartbeater.cc:511] Master 127.0.98.62:32893 requested a full tablet report, sending...
I20260620 14:29:24.328157 15792 ts_manager.cc:194] Registered new tserver with Master: 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145)
I20260620 14:29:24.328670 15792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38243
I20260620 14:29:24.328776 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.320849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.0.98.62:32893)
0620 14:29:24.321181 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:24.321183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:24.321189 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:29:24.321209 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:24.321209 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:24.321219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:24.321295 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:24.322005 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.322566 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.323504 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.323718 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.323810 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.325473 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:24.325474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:24.325475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:24.325482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:24.327342 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:24.327737 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:24.327738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:24.327738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:24.327795 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:24.328039 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:24.328040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:24.328041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:24.328174 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:24.328346 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:24.328435 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:29:24.328644 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":102,"threads_started":1}
I20260620 14:29:24.329569 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:29:24.332307 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.329970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.0.98.62:32893)
0620 14:29:24.330037 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:24.330039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:24.330145 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:29:24.330203 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:24.330204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:24.330232 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:24.330299 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:29:24.330478 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.330917 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.331706 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.331891 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.331963 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.332107 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:29:24.332146 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 14:29:24.332262 (+ 116us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50342: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 14:29:24.332381 16254 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50342: token expired
W20260620 14:29:24.332499 15722 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:32893)
I20260620 14:29:24.338239 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.332792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.0.98.62:32893)
0620 14:29:24.332842 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:24.332844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:24.332851 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:29:24.332906 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:24.332906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:24.332914 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:24.332954 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:24.333388 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.333854 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.334610 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.334794 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.334849 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.335973 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:24.335974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:24.335975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:24.335980 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:24.337248 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:24.337548 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:24.337549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:24.337549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:24.337596 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:24.337748 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:24.337749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:24.337749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:24.337838 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:24.337912 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:24.338074 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:29:24.338136 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:29:24.339285 15792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:29:24.340915 15792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:29:24.343070 15914 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec9fa7dfe99498885ac9cef850e2420 (DEFAULT_TABLE table=test-table [id=efbf307a57444004bae3be01cdc0de5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:24.343278 15914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec9fa7dfe99498885ac9cef850e2420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:24.344585 15872 tablet_bootstrap.cc:492] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: Bootstrap starting.
I20260620 14:29:24.344933 15872 tablet_bootstrap.cc:654] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:24.345530 15872 tablet_bootstrap.cc:492] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: No bootstrap required, opened a new log
I20260620 14:29:24.345585 15872 ts_tablet_manager.cc:1397] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:24.345919 15872 raft_consensus.cc:348] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:24.345999 15872 raft_consensus.cc:374] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:24.346040 15872 raft_consensus.cc:729] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae2ad84973e4f45b807a4a19e5b69f0, State: Initialized, Role: FOLLOWER
I20260620 14:29:24.346086 15872 consensus_queue.cc:260] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:24.346166 15872 ts_tablet_manager.cc:1428] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:24.346676 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.342857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34879 (local address 127.0.0.1:58138)
0620 14:29:24.342892 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:24.342903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:24.342938 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:24.343498 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:24.343499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:24.343502 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:24.343548 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.343549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.344427 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.344429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.345540 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.345546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.346411 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.346412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.346485 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.346504 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:29:24.346559 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:29:24.346606 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 14:29:24.346692 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.343049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.0.98.2:34879)
0620 14:29:24.343235 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:24.343240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:24.343251 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:29:24.343291 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:24.343293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:24.343308 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:24.343490 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:24.343619 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.344330 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.345660 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.346271 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.346387 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.346456 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:24.346572 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 14:29:24.346614 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20260620 14:29:24.347405 16052 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec9fa7dfe99498885ac9cef850e2420 (DEFAULT_TABLE table=test-table [id=efbf307a57444004bae3be01cdc0de5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:24.347765 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec9fa7dfe99498885ac9cef850e2420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:24.348534 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.342977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.0.98.3:36145)
0620 14:29:24.343279 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:24.343284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:24.344882 (+ 1598us) server_negotiation.cc:407] Connection header received
0620 14:29:24.344942 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:24.344944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:24.344964 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:24.345042 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:24.345244 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.346328 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.347630 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.348194 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.348255 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.348291 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:24.348443 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 14:29:24.348472 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":102,"threads_started":1}
I20260620 14:29:24.348532 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.342857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36145 (local address 127.0.0.1:56286)
0620 14:29:24.344803 (+ 1946us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:24.344812 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:24.344869 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:24.345130 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:24.345131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:24.345134 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:24.345177 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.345179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.346442 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.346443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.347511 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.347514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.348279 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.348280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.348356 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.348361 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:29:24.348426 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:29:24.348464 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1930,"mutex_wait_us":123,"thread_start_us":95,"threads_started":1}
I20260620 14:29:24.349226 16184 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec9fa7dfe99498885ac9cef850e2420 (DEFAULT_TABLE table=test-table [id=efbf307a57444004bae3be01cdc0de5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:24.349555 16184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec9fa7dfe99498885ac9cef850e2420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:24.351400 16260 tablet_bootstrap.cc:492] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7: Bootstrap starting.
I20260620 14:29:24.351992 16260 tablet_bootstrap.cc:654] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:24.352629 16260 log.cc:824] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:24.353034 16262 tablet_bootstrap.cc:492] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147: Bootstrap starting.
I20260620 14:29:24.353374 16260 tablet_bootstrap.cc:492] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7: No bootstrap required, opened a new log
I20260620 14:29:24.353428 16260 ts_tablet_manager.cc:1397] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:24.353655 16262 tablet_bootstrap.cc:654] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:24.354149 16262 log.cc:824] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:24.354905 16262 tablet_bootstrap.cc:492] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147: No bootstrap required, opened a new log
I20260620 14:29:24.354985 16262 ts_tablet_manager.cc:1397] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:29:24.354909 16260 raft_consensus.cc:348] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:24.355078 16260 raft_consensus.cc:374] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:24.355124 16260 raft_consensus.cc:729] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b87b7777e74e9c9bcee013eb506ca7, State: Initialized, Role: FOLLOWER
I20260620 14:29:24.355235 16260 consensus_queue.cc:260] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:24.355497 16260 ts_tablet_manager.cc:1428] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:24.355533 16118 heartbeater.cc:503] Master 127.0.98.62:32893 was elected leader, sending a full tablet report...
I20260620 14:29:24.356949 16262 raft_consensus.cc:348] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:24.357101 16262 raft_consensus.cc:374] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:24.357148 16262 raft_consensus.cc:729] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0452373f42024dbcb07d1840c5075147, State: Initialized, Role: FOLLOWER
I20260620 14:29:24.357257 16262 consensus_queue.cc:260] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:24.357517 16262 ts_tablet_manager.cc:1428] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:29:24.357590 16250 heartbeater.cc:503] Master 127.0.98.62:32893 was elected leader, sending a full tablet report...
W20260620 14:29:24.431838 16119 tablet.cc:2369] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:29:24.547515 15981 tablet.cc:2369] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:29:24.566991 16251 tablet.cc:2369] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:29:24.578408 15986 raft_consensus.cc:482] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:24.578552 15986 raft_consensus.cc:504] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:24.578889 15986 leader_election.cc:288] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145), 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879)
I20260620 14:29:24.583204 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.579313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38559 (local address 127.0.98.2:34879)
0620 14:29:24.579354 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:24.579356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:24.579441 (+ 85us) server_negotiation.cc:407] Connection header received
0620 14:29:24.579555 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:24.579556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:24.579570 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:24.579635 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:24.579848 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.580594 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.582199 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.582801 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.582876 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.582887 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:24.583092 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 14:29:24.583131 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:29:24.583204 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.579298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59893 (local address 127.0.98.3:36145)
0620 14:29:24.579348 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:24.579350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:24.579495 (+ 145us) server_negotiation.cc:407] Connection header received
0620 14:29:24.579564 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:24.579566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:24.579578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:24.579635 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:24.579848 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.580590 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.582201 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.582801 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.582876 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.582887 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:24.583092 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 14:29:24.583131 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:29:24.583235 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.579220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36145 (local address 127.0.98.1:59893)
0620 14:29:24.579363 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:24.579427 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:24.579482 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:24.579680 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:24.579681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:24.579685 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:24.579754 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.579757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.580685 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.580686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.582102 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.582105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.582896 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.582898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.582969 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.582999 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:29:24.583066 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:29:24.583130 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":125,"thread_start_us":61,"threads_started":1}
I20260620 14:29:24.583235 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.579220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34879 (local address 127.0.98.1:38559)
0620 14:29:24.579363 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:24.579375 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:24.579461 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:24.579680 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:24.579681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:24.579685 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:24.579771 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.579773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.580684 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.580685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.582102 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:24.582105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.582894 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.582909 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:24.582974 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.583001 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:29:24.583066 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:29:24.583129 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":106,"thread_start_us":61,"threads_started":1}
I20260620 14:29:24.583658 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec9fa7dfe99498885ac9cef850e2420" candidate_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b87b7777e74e9c9bcee013eb506ca7" is_pre_election: true
I20260620 14:29:24.583822 16072 raft_consensus.cc:2454] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae2ad84973e4f45b807a4a19e5b69f0 in term 0.
I20260620 14:29:24.583777 16204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec9fa7dfe99498885ac9cef850e2420" candidate_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0452373f42024dbcb07d1840c5075147" is_pre_election: true
I20260620 14:29:24.583948 16204 raft_consensus.cc:2454] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae2ad84973e4f45b807a4a19e5b69f0 in term 0.
I20260620 14:29:24.584100 15867 leader_election.cc:302] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93b87b7777e74e9c9bcee013eb506ca7, bae2ad84973e4f45b807a4a19e5b69f0; no voters:
I20260620 14:29:24.584250 15986 raft_consensus.cc:2781] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:29:24.584312 15986 raft_consensus.cc:482] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:29:24.584370 15986 raft_consensus.cc:3037] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:24.585104 15986 raft_consensus.cc:504] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:24.585253 15986 leader_election.cc:288] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 1 election: Requested vote from peers 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145), 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879)
I20260620 14:29:24.585421 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec9fa7dfe99498885ac9cef850e2420" candidate_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b87b7777e74e9c9bcee013eb506ca7"
I20260620 14:29:24.585453 16204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec9fa7dfe99498885ac9cef850e2420" candidate_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0452373f42024dbcb07d1840c5075147"
I20260620 14:29:24.585513 16204 raft_consensus.cc:3037] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:24.585527 16072 raft_consensus.cc:3037] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:24.586257 16204 raft_consensus.cc:2454] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae2ad84973e4f45b807a4a19e5b69f0 in term 1.
I20260620 14:29:24.586266 16072 raft_consensus.cc:2454] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae2ad84973e4f45b807a4a19e5b69f0 in term 1.
I20260620 14:29:24.586449 15869 leader_election.cc:302] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0452373f42024dbcb07d1840c5075147, bae2ad84973e4f45b807a4a19e5b69f0; no voters:
I20260620 14:29:24.586562 15986 raft_consensus.cc:2781] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:24.586719 15986 raft_consensus.cc:686] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 LEADER]: Becoming Leader. State: Replica: bae2ad84973e4f45b807a4a19e5b69f0, State: Running, Role: LEADER
I20260620 14:29:24.586804 15986 consensus_queue.cc:237] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:24.587607 15792 catalog_manager.cc:5495] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to bae2ad84973e4f45b807a4a19e5b69f0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } health_report { overall_health: HEALTHY } } }
I20260620 14:29:24.630852 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.628097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.0.98.1:36811)
0620 14:29:24.628290 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:24.628292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:24.628299 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:29:24.628337 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:24.628338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:24.628356 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:24.628482 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:29:24.628641 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.629178 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.629968 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:24.630184 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:24.630267 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:24.630486 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:29:24.630590 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:29:24.630677 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:24.630718 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 14:29:24.630740 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":112,"threads_started":1}
I20260620 14:29:24.632686 16204 raft_consensus.cc:1261] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147 [term 1 FOLLOWER]: Refusing update from remote peer bae2ad84973e4f45b807a4a19e5b69f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:24.632697 16072 raft_consensus.cc:1261] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7 [term 1 FOLLOWER]: Refusing update from remote peer bae2ad84973e4f45b807a4a19e5b69f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:24.633107 16268 consensus_queue.cc:1035] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:24.633257 15986 consensus_queue.cc:1035] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:24.635850 16271 mvcc.cc:205] Tried to move back new op lower bound from 7298931771932585984 to 7298931771748188160. Current Snapshot: MvccSnapshot[applied={T|T < 7298931771932585984}]
I20260620 14:29:24.636335 16272 mvcc.cc:205] Tried to move back new op lower bound from 7298931771932585984 to 7298931771748188160. Current Snapshot: MvccSnapshot[applied={T|T < 7298931771932585984}]
I20260620 14:29:24.637842 15791 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:24.637954 15791 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:24.638768 15791 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Sending DeleteTablet for 3 replicas of tablet 1ec9fa7dfe99498885ac9cef850e2420
I20260620 14:29:24.639114 16184 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ec9fa7dfe99498885ac9cef850e2420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56286
I20260620 14:29:24.639142 15914 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ec9fa7dfe99498885ac9cef850e2420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39208
I20260620 14:29:24.639120 16052 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ec9fa7dfe99498885ac9cef850e2420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58138
I20260620 14:29:24.639334 16283 tablet_replica.cc:330] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7: stopping tablet replica
I20260620 14:29:24.639436 16283 raft_consensus.cc:2229] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:24.639534 16281 tablet_replica.cc:330] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: stopping tablet replica
I20260620 14:29:24.639551 16283 raft_consensus.cc:2258] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:24.639570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 15830
I20260620 14:29:24.640252 16282 tablet_replica.cc:330] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147: stopping tablet replica
I20260620 14:29:24.640357 16282 raft_consensus.cc:2229] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:24.640455 16282 raft_consensus.cc:2258] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:24.640980 16283 ts_tablet_manager.cc:1905] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:24.642308 16282 ts_tablet_manager.cc:1905] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:24.642652 16283 ts_tablet_manager.cc:1918] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:24.642725 16283 log.cc:1196] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/1ec9fa7dfe99498885ac9cef850e2420
I20260620 14:29:24.643018 16283 ts_tablet_manager.cc:1939] T 1ec9fa7dfe99498885ac9cef850e2420 P 93b87b7777e74e9c9bcee013eb506ca7: Deleting consensus metadata
I20260620 14:29:24.643419 15777 catalog_manager.cc:4841] TS 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879): tablet 1ec9fa7dfe99498885ac9cef850e2420 (table test-table [id=efbf307a57444004bae3be01cdc0de5c]) successfully deleted
I20260620 14:29:24.643805 16282 ts_tablet_manager.cc:1918] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:24.643893 16282 log.cc:1196] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/1ec9fa7dfe99498885ac9cef850e2420
I20260620 14:29:24.644138 16282 ts_tablet_manager.cc:1939] T 1ec9fa7dfe99498885ac9cef850e2420 P 0452373f42024dbcb07d1840c5075147: Deleting consensus metadata
I20260620 14:29:24.644534 15779 catalog_manager.cc:4841] TS 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145): tablet 1ec9fa7dfe99498885ac9cef850e2420 (table test-table [id=efbf307a57444004bae3be01cdc0de5c]) successfully deleted
W20260620 14:29:24.645715 15776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36811 (error 108)
I20260620 14:29:24.646078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 15984
I20260620 14:29:24.646168 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.646038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36811 (local address 127.0.0.1:39218)
0620 14:29:24.646093 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:24.646128 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:29:24.647969 15776 catalog_manager.cc:4584] TS bae2ad84973e4f45b807a4a19e5b69f0 (127.0.98.1:36811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ec9fa7dfe99498885ac9cef850e2420: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36811: connect: Connection refused (error 111)
I20260620 14:29:24.651031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 16121
I20260620 14:29:24.655604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 15759
I20260620 14:29:44.661118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32893
--webserver_interface=127.0.98.62
--webserver_port=34985
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:44.743996 16285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:44.744294 16285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:44.744338 16285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:44.745512 16285 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:29:44.745576 16285 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:29:44.745597 16285 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:29:44.745617 16285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:44.745648 16285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:44.745678 16285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:44.745702 16285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:44.745736 16285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:44.747243 16285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:29:44.747529 16285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:44.747766 16285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:44.753064 16290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:44.753057 16291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:44.753074 16285 server_base.cc:1034] running on GCE node
W20260620 14:29:44.753057 16293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:44.753672 16285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:44.753980 16285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:44.755147 16285 hybrid_clock.cc:648] HybridClock initialized: now 1781965784755145 us; error 41 us; skew 500 ppm
Jun 20 14:29:44 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:44.757814 16285 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:29:44.758282 16285 webserver.cc:460] Webserver started at http://127.0.98.62:34985/ using document root <none> and password file <none>
I20260620 14:29:44.758570 16285 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:44.758620 16285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:44.760115 16285 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:44.760907 16300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:44.761071 16285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:44.761164 16285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "8296c81c74724ea099cae5a5ca8770cc"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "608f7d57c5bb931be9f9fdf014eea33f"
server_key_iv: "2ac5acae09f2feee294cda6708af5241"
server_key_version: "encryptionkey@0"
I20260620 14:29:44.761534 16285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:44.772655 16285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:44.774910 16285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:44.775030 16285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:44.779071 16285 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32893
I20260620 14:29:44.779203 16352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32893 every 8 connection(s)
I20260620 14:29:44.779536 16285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:44.781081 16353 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:29:44.781795 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Bootstrap starting.
I20260620 14:29:44.786653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 16285
I20260620 14:29:44.787038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36811
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32893
--webserver_port=38795
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:29:44.790835 16353 log.cc:824] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:44.793134 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:29:44.793483 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: Bootstrap complete.
I20260620 14:29:44.795850 16353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260620 14:29:44.796074 16353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8296c81c74724ea099cae5a5ca8770cc, State: Initialized, Role: FOLLOWER
I20260620 14:29:44.796252 16353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260620 14:29:44.796319 16353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:44.796386 16353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:44.796432 16353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:29:44.797482 16353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260620 14:29:44.797573 16353 leader_election.cc:302] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8296c81c74724ea099cae5a5ca8770cc; no voters:
I20260620 14:29:44.797807 16353 leader_election.cc:288] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:29:44.797843 16357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:29:44.797963 16357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [term 3 LEADER]: Becoming Leader. State: Replica: 8296c81c74724ea099cae5a5ca8770cc, State: Running, Role: LEADER
I20260620 14:29:44.798061 16357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260620 14:29:44.798091 16353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:44.798382 16358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8296c81c74724ea099cae5a5ca8770cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } } }
I20260620 14:29:44.798537 16358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:44.798828 16359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8296c81c74724ea099cae5a5ca8770cc. Latest consensus state: current_term: 3 leader_uuid: "8296c81c74724ea099cae5a5ca8770cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296c81c74724ea099cae5a5ca8770cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } } }
I20260620 14:29:44.798924 16359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:44.800071 16373 catalog_manager.cc:1259] Loaded cluster ID: 997be8ca6dd64a1cbb878514405073f2
I20260620 14:29:44.800138 16373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: loading cluster ID for follower catalog manager: success
I20260620 14:29:44.800936 16373 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8296c81c74724ea099cae5a5ca8770cc: acquiring CA information for follower catalog manager: success
F20260620 14:29:44.801357 16373 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f812f92f9ad google::LogMessage::Fail() at ??:0
@ 0x7f812f93288a google::LogMessage::SendToLog() at ??:0
@ 0x7f812f92f4de google::LogMessage::Flush() at ??:0
@ 0x7f812f930a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f813209c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f81378886b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f8137888989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f8137888e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f813788b8c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f81317f2954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f81326566db start_thread at ??:0
@ 0x7f812e42d61f clone at ??:0
W20260620 14:29:44.905445 16355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:44.905680 16355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:44.905709 16355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:44.907058 16355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:44.907107 16355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:44.907137 16355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:29:44.908685 16355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38795
--tserver_master_addrs=127.0.98.62:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:29:44.908947 16355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:44.909207 16355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:44.915136 16392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:44.915131 16391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:44.915169 16394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:44.915578 16355 server_base.cc:1034] running on GCE node
I20260620 14:29:44.915805 16355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:44.916107 16355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:44.917313 16355 hybrid_clock.cc:648] HybridClock initialized: now 1781965784917282 us; error 58 us; skew 500 ppm
Jun 20 14:29:44 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:44.920511 16355 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:29:44.921115 16355 webserver.cc:460] Webserver started at http://127.0.98.1:38795/ using document root <none> and password file <none>
I20260620 14:29:44.921396 16355 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:44.921465 16355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:44.923345 16355 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:44.924149 16401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:44.924356 16355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:44.924453 16355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "bae2ad84973e4f45b807a4a19e5b69f0"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "1045301dea044da71ed2b177820ec80b"
server_key_iv: "c2425c2117cb5bed16ece342b5d39b9d"
server_key_version: "encryptionkey@0"
I20260620 14:29:44.924809 16355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:44.945199 16355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:44.948537 16355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:44.948706 16355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:44.948953 16355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:44.949573 16408 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:29:44.951494 16355 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:29:44.951646 16355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:29:44.951771 16355 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:29:44.952680 16355 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:29:44.952797 16355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:29:44.952806 16408 tablet_bootstrap.cc:492] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: Bootstrap starting.
I20260620 14:29:44.960388 16355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36811
I20260620 14:29:44.960942 16355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:29:44.962558 16515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36811 every 8 connection(s)
I20260620 14:29:44.963748 16408 log.cc:824] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:44.964666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 16355
I20260620 14:29:44.964986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34879
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32893
--webserver_port=36567
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:29:44.966943 16408 tablet_bootstrap.cc:492] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:29:44.967275 16408 tablet_bootstrap.cc:492] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: Bootstrap complete.
I20260620 14:29:44.968243 16408 ts_tablet_manager.cc:1397] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20260620 14:29:44.969318 16408 raft_consensus.cc:348] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:44.969575 16408 raft_consensus.cc:729] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae2ad84973e4f45b807a4a19e5b69f0, State: Initialized, Role: FOLLOWER
I20260620 14:29:44.969772 16408 consensus_queue.cc:260] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:44.970093 16408 ts_tablet_manager.cc:1428] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260620 14:29:45.021092 16518 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:29:44.961770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:60497)
0620 14:29:44.961995 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:44.962048 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:44.962764 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:45.020883 (+ 58119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":107,"threads_started":1}
W20260620 14:29:45.021433 16516 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32893 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:32893: Client connection negotiation failed: client connection to 127.0.98.62:32893: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:29:45.021821 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:45.021692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:49831)
0620 14:29:45.021735 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:45.021792 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 14:29:45.022167 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:45.022073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:57739)
0620 14:29:45.022120 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:45.022149 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:29:45.022298 16516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:29:45.053934 16520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:45.054152 16520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:45.054177 16520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:45.055392 16520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:45.055444 16520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:45.055472 16520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:29:45.056973 16520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36567
--tserver_master_addrs=127.0.98.62:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:29:45.057240 16520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:45.057482 16520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:45.062704 16526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:45.062682 16527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:45.062686 16529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:45.062945 16520 server_base.cc:1034] running on GCE node
I20260620 14:29:45.063148 16520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:45.063411 16520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:45.064579 16520 hybrid_clock.cc:648] HybridClock initialized: now 1781965785064560 us; error 31 us; skew 500 ppm
Jun 20 14:29:45 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:45.067138 16520 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:29:45.067536 16520 webserver.cc:460] Webserver started at http://127.0.98.2:36567/ using document root <none> and password file <none>
I20260620 14:29:45.067734 16520 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:45.067775 16520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:45.069101 16520 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:45.069737 16536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:45.069945 16520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:45.070031 16520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "93b87b7777e74e9c9bcee013eb506ca7"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "251b237e518ad54c60543337fd7b5445"
server_key_iv: "50f361390bfa851bb5edce9aebd5de27"
server_key_version: "encryptionkey@0"
I20260620 14:29:45.070444 16520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:45.088238 16520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:45.091172 16520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:45.091295 16520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:45.091516 16520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:45.091883 16520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:45.091921 16520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:45.091979 16520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:45.092010 16520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:45.098444 16520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34879
I20260620 14:29:45.098641 16649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34879 every 8 connection(s)
I20260620 14:29:45.098901 16520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:29:45.099993 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:45.099581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:60913)
0620 14:29:45.099738 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:45.099813 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
W20260620 14:29:45.100466 16650 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32893 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:32893: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
I20260620 14:29:45.100766 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:45.100678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:48697)
0620 14:29:45.100721 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:45.100749 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 14:29:45.101058 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:45.100983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:46399)
0620 14:29:45.101023 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:45.101044 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:29:45.101172 16650 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:29:45.103634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 16520
I20260620 14:29:45.103947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36145
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32893
--webserver_port=38349
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:45.187783 16653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:45.187996 16653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:45.188020 16653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:45.189254 16653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:45.189307 16653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:45.189335 16653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:29:45.190853 16653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38349
--tserver_master_addrs=127.0.98.62:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:29:45.191094 16653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:45.191344 16653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:45.196512 16659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:45.196497 16658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:45.196684 16653 server_base.cc:1034] running on GCE node
W20260620 14:29:45.196497 16661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:45.196962 16653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:45.197214 16653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:45.198508 16653 hybrid_clock.cc:648] HybridClock initialized: now 1781965785198407 us; error 36 us; skew 500 ppm
Jun 20 14:29:45 dist-test-slave-hfc3 krb5kdc[15213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:45.201179 16653 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:29:45.201651 16653 webserver.cc:460] Webserver started at http://127.0.98.3:38349/ using document root <none> and password file <none>
I20260620 14:29:45.201879 16653 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:45.201936 16653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:45.203346 16653 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:29:45.203974 16668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:45.204147 16653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:29:45.204252 16653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "0452373f42024dbcb07d1840c5075147"
format_stamp: "Formatted at 2026-06-20 14:29:03 on dist-test-slave-hfc3"
server_key: "fb7a42fcfad8a32ecceb62940de284c8"
server_key_iv: "721bd0bcaef9e3763131a933e346eb81"
server_key_version: "encryptionkey@0"
I20260620 14:29:45.204605 16653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:45.214304 16653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:45.216308 16653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:45.216475 16653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:45.216697 16653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:45.217007 16653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:45.217072 16653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:45.217114 16653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:45.217142 16653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:45.223199 16653 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36145
I20260620 14:29:45.223248 16781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36145 every 8 connection(s)
I20260620 14:29:45.223591 16653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:29:45.224603 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:45.224211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:45477)
0620 14:29:45.224373 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:45.224465 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
W20260620 14:29:45.224994 16782 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32893 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:32893: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
I20260620 14:29:45.225255 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:45.225166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:40239)
0620 14:29:45.225213 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:45.225238 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:29:45.225507 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:45.225455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:37205)
0620 14:29:45.225476 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:45.225494 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 14:29:45.225615 16782 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:29:45.229211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 16653
I20260620 14:29:46.022857 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:46.022659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:37971)
0620 14:29:46.022775 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:46.022811 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260620 14:29:46.101715 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:46.101493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:49173)
0620 14:29:46.101632 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:46.101670 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":58,"threads_started":1}
I20260620 14:29:46.226199 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:46.225931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:57611)
0620 14:29:46.226106 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:46.226145 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260620 14:29:46.805505 16789 raft_consensus.cc:482] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:46.805634 16789 raft_consensus.cc:504] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:46.805954 16789 leader_election.cc:288] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145), 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879)
Jun 20 14:29:46 dist-test-slave-hfc3 krb5kdc[15213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 14:29:46 dist-test-slave-hfc3 krb5kdc[15213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 14:29:46.821834 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:46.806172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34879 (local address 127.0.98.1:58185)
0620 14:29:46.806325 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:46.806337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:46.806414 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:46.807185 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:46.807188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:46.807611 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:46.807788 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:46.807794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:46.808881 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:46.808882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:46.809626 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:46.809627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:46.810234 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:46.810235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:46.810378 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:46.816189 (+ 5811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:46.816195 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:46.818217 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:46.820300 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:46.820302 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:46.820302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:46.820529 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:46.820825 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:46.820826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:46.820826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:46.820903 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:46.821370 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:46.821370 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:46.821433 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 14:29:46.821516 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:29:46.821721 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260620 14:29:46.821959 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:46.806119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36145 (local address 127.0.98.1:44275)
0620 14:29:46.806249 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:46.806262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:46.806365 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:46.807229 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:46.807230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:46.807593 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:46.807798 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:46.807801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:46.809032 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:46.809033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:46.809627 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:46.809629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:46.810263 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:46.810264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:46.810377 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:46.816179 (+ 5802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:46.816188 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:46.817754 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:46.820104 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:46.820111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:46.820113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:46.820418 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:46.820696 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:46.820697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:46.820698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:46.820833 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:46.821266 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:46.821274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:46.821434 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 14:29:46.821558 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 14:29:46.821816 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":114,"thread_start_us":52,"threads_started":1}
I20260620 14:29:46.821897 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:46.806245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44275 (local address 127.0.98.3:36145)
0620 14:29:46.806450 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:46.806471 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:46.806491 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:29:46.807033 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:46.807036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:46.807078 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:46.807231 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:46.807924 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:46.808937 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:46.809736 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:46.810159 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:46.810296 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:46.817883 (+ 7587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:46.817888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:46.817888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:46.817901 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:46.820006 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:46.820497 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:46.820501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:46.820502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:46.820622 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:46.820908 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:46.820909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:46.820910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:46.821182 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:46.821244 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:46.821627 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 14:29:46.821786 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
I20260620 14:29:46.822284 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:46.806306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58185 (local address 127.0.98.2:34879)
0620 14:29:46.806487 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:46.806505 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:46.806521 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:29:46.806942 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:46.806944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:46.806985 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:46.807142 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:46.807915 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:46.808776 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:46.809736 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:46.810132 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:46.810249 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:46.818386 (+ 8137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:46.818391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:46.818391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:46.818411 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:46.820205 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:46.820615 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:46.820618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:46.820618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:46.820749 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:46.820987 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:46.820988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:46.820989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:46.821286 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:46.821378 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:46.821605 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:29:46.821793 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
W20260620 14:29:46.822834 16405 leader_election.cc:341] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145): Not found: Tablet not found: 1ec9fa7dfe99498885ac9cef850e2420
W20260620 14:29:46.823096 16403 leader_election.cc:341] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879): Not found: Tablet not found: 1ec9fa7dfe99498885ac9cef850e2420
I20260620 14:29:46.823148 16403 leader_election.cc:302] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bae2ad84973e4f45b807a4a19e5b69f0; no voters: 0452373f42024dbcb07d1840c5075147, 93b87b7777e74e9c9bcee013eb506ca7
I20260620 14:29:46.823261 16789 raft_consensus.cc:2726] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:29:47.023630 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:47.023420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:59201)
0620 14:29:47.023511 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:47.023557 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 14:29:47.102541 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:47.102214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:46457)
0620 14:29:47.102434 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:47.102494 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260620 14:29:47.226922 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:47.226658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:47979)
0620 14:29:47.226819 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:47.226873 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260620 14:29:48.024451 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:48.024207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:46109)
0620 14:29:48.024354 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:48.024402 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260620 14:29:48.103201 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:48.102980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:36405)
0620 14:29:48.103109 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:48.103144 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260620 14:29:48.227687 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:48.227449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:48845)
0620 14:29:48.227609 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:48.227649 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260620 14:29:49.021282 16799 raft_consensus.cc:482] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:49.021382 16799 raft_consensus.cc:504] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:49.021540 16799 leader_election.cc:288] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145), 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879)
W20260620 14:29:49.022032 16405 leader_election.cc:341] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145): Not found: Tablet not found: 1ec9fa7dfe99498885ac9cef850e2420
W20260620 14:29:49.022099 16403 leader_election.cc:341] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879): Not found: Tablet not found: 1ec9fa7dfe99498885ac9cef850e2420
I20260620 14:29:49.022125 16405 leader_election.cc:302] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bae2ad84973e4f45b807a4a19e5b69f0; no voters: 0452373f42024dbcb07d1840c5075147, 93b87b7777e74e9c9bcee013eb506ca7
I20260620 14:29:49.022197 16799 raft_consensus.cc:2726] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:29:49.025148 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:49.024960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:50035)
0620 14:29:49.025078 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:49.025112 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260620 14:29:49.103888 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:49.103668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:42779)
0620 14:29:49.103790 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:49.103830 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260620 14:29:49.228344 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:49.228106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:40245)
0620 14:29:49.228253 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:49.228289 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
I20260620 14:29:50.025893 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:50.025659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:53285)
0620 14:29:50.025808 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:50.025849 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260620 14:29:50.104610 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:50.104355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:55563)
0620 14:29:50.104531 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:50.104568 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":121,"threads_started":1}
I20260620 14:29:50.229031 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:50.228841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:52663)
0620 14:29:50.228955 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:50.228991 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260620 14:29:50.676865 16806 raft_consensus.cc:482] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:50.676960 16806 raft_consensus.cc:504] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:50.677172 16806 leader_election.cc:288] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145), 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879)
W20260620 14:29:50.677578 16405 leader_election.cc:341] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145): Not found: Tablet not found: 1ec9fa7dfe99498885ac9cef850e2420
W20260620 14:29:50.677670 16403 leader_election.cc:341] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879): Not found: Tablet not found: 1ec9fa7dfe99498885ac9cef850e2420
I20260620 14:29:50.677755 16403 leader_election.cc:302] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bae2ad84973e4f45b807a4a19e5b69f0; no voters: 0452373f42024dbcb07d1840c5075147, 93b87b7777e74e9c9bcee013eb506ca7
I20260620 14:29:50.677820 16806 raft_consensus.cc:2726] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:29:51.026703 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:51.026484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:35983)
0620 14:29:51.026615 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:51.026656 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260620 14:29:51.105340 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:51.105092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:42229)
0620 14:29:51.105259 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:51.105293 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260620 14:29:51.229753 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:51.229483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:34117)
0620 14:29:51.229661 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:51.229708 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260620 14:29:52.027343 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:52.027140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:41525)
0620 14:29:52.027267 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:52.027300 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260620 14:29:52.106082 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:52.105851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:37037)
0620 14:29:52.105999 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:52.106038 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260620 14:29:52.230501 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:52.230191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:35805)
0620 14:29:52.230395 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:52.230434 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":59,"threads_started":1}
I20260620 14:29:52.734508 16813 raft_consensus.cc:482] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:52.734602 16813 raft_consensus.cc:504] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:52.734781 16813 leader_election.cc:288] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145), 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879)
W20260620 14:29:52.735134 16405 leader_election.cc:341] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145): Not found: Tablet not found: 1ec9fa7dfe99498885ac9cef850e2420
W20260620 14:29:52.735224 16403 leader_election.cc:341] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879): Not found: Tablet not found: 1ec9fa7dfe99498885ac9cef850e2420
I20260620 14:29:52.735301 16403 leader_election.cc:302] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bae2ad84973e4f45b807a4a19e5b69f0; no voters: 0452373f42024dbcb07d1840c5075147, 93b87b7777e74e9c9bcee013eb506ca7
I20260620 14:29:52.735387 16813 raft_consensus.cc:2726] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:29:53.028195 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:53.027967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:35185)
0620 14:29:53.028101 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:53.028149 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260620 14:29:53.106935 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:53.106694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:33429)
0620 14:29:53.106853 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:53.106893 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260620 14:29:53.231355 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:53.231028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:42925)
0620 14:29:53.231237 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:53.231297 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
I20260620 14:29:54.028975 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:54.028656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:35277)
0620 14:29:54.028878 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:54.028924 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
I20260620 14:29:54.107728 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:54.107494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:59547)
0620 14:29:54.107633 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:54.107684 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260620 14:29:54.232180 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:54.231920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:55541)
0620 14:29:54.232082 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:54.232131 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":69,"threads_started":1}
I20260620 14:29:54.601255 16820 raft_consensus.cc:482] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:54.601359 16820 raft_consensus.cc:504] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:54.601540 16820 leader_election.cc:288] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145), 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879)
W20260620 14:29:54.601989 16403 leader_election.cc:341] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879): Not found: Tablet not found: 1ec9fa7dfe99498885ac9cef850e2420
W20260620 14:29:54.602084 16405 leader_election.cc:341] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145): Not found: Tablet not found: 1ec9fa7dfe99498885ac9cef850e2420
I20260620 14:29:54.602154 16405 leader_election.cc:302] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bae2ad84973e4f45b807a4a19e5b69f0; no voters: 0452373f42024dbcb07d1840c5075147, 93b87b7777e74e9c9bcee013eb506ca7
I20260620 14:29:54.602248 16820 raft_consensus.cc:2726] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:29:55.029726 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:55.029512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:56439)
0620 14:29:55.029637 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:55.029676 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260620 14:29:55.108532 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:55.108263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:33235)
0620 14:29:55.108394 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:55.108432 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260620 14:29:55.232947 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:55.232699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:43735)
0620 14:29:55.232864 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:55.232904 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260620 14:29:56.030964 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:56.030538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:47323)
0620 14:29:56.030775 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:56.030867 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":102,"threads_started":1}
I20260620 14:29:56.109750 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:56.109345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:41407)
0620 14:29:56.109576 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:56.109667 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
I20260620 14:29:56.234123 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:56.233666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:45963)
0620 14:29:56.233961 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:56.234040 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":218,"threads_started":1}
I20260620 14:29:57.031848 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:57.031594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:54355)
0620 14:29:57.031761 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:57.031803 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260620 14:29:57.110728 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:57.110519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:34973)
0620 14:29:57.110637 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:57.110679 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260620 14:29:57.235033 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:57.234801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:49315)
0620 14:29:57.234951 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:57.234988 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260620 14:29:57.825100 16830 raft_consensus.cc:482] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:57.825215 16830 raft_consensus.cc:504] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452373f42024dbcb07d1840c5075147" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36145 } } peers { permanent_uuid: "93b87b7777e74e9c9bcee013eb506ca7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "bae2ad84973e4f45b807a4a19e5b69f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36811 } }
I20260620 14:29:57.825441 16830 leader_election.cc:288] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145), 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879)
W20260620 14:29:57.825817 16403 leader_election.cc:341] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 93b87b7777e74e9c9bcee013eb506ca7 (127.0.98.2:34879): Not found: Tablet not found: 1ec9fa7dfe99498885ac9cef850e2420
W20260620 14:29:57.825901 16405 leader_election.cc:341] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0452373f42024dbcb07d1840c5075147 (127.0.98.3:36145): Not found: Tablet not found: 1ec9fa7dfe99498885ac9cef850e2420
I20260620 14:29:57.825990 16405 leader_election.cc:302] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bae2ad84973e4f45b807a4a19e5b69f0; no voters: 0452373f42024dbcb07d1840c5075147, 93b87b7777e74e9c9bcee013eb506ca7
I20260620 14:29:57.826079 16830 raft_consensus.cc:2726] T 1ec9fa7dfe99498885ac9cef850e2420 P bae2ad84973e4f45b807a4a19e5b69f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:29:58.032536 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:58.032314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:40947)
0620 14:29:58.032448 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:58.032490 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260620 14:29:58.111405 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:58.111179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:43849)
0620 14:29:58.111308 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:58.111346 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260620 14:29:58.235728 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:58.235499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:33329)
0620 14:29:58.235631 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:58.235668 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260620 14:29:59.033389 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:59.033104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:50697)
0620 14:29:59.033281 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:59.033338 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
I20260620 14:29:59.112226 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:59.111894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:35085)
0620 14:29:59.112025 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:59.112124 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260620 14:29:59.236398 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:59.236195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:47899)
0620 14:29:59.236319 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:59.236355 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260620 14:30:00.034202 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:00.033997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:35485)
0620 14:30:00.034127 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:00.034159 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260620 14:30:00.113201 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:00.112937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:53255)
0620 14:30:00.113098 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:00.113139 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260620 14:30:00.233873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 16355
I20260620 14:30:00.237135 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:00.236933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:35009)
0620 14:30:00.237061 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:00.237093 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260620 14:30:00.239965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 16520
I20260620 14:30:00.244808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 16653
2026-06-20T14:30:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58462 ms)
[----------] 30 tests from SecurityITest (346155 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:00 dist-test-slave-hfc3 krb5kdc[16842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:00 dist-test-slave-hfc3 krb5kdc[16842](info): set up 2 sockets
Jun 20 14:30:00 dist-test-slave-hfc3 krb5kdc[16842](info): commencing operation
krb5kdc: starting...
W20260620 14:30:01.290545 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:01 dist-test-slave-hfc3 krb5kdc[16842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:01.417644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39051
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:01.502732 16855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:01.503005 16855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:01.503044 16855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:01.504282 16855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:01.504371 16855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:01.504400 16855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:01.504433 16855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:01.504457 16855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:01.506029 16855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:01.506388 16855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:01.506656 16855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:01.512373 16860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:01.512346 16863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:01.512351 16861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:01.512388 16855 server_base.cc:1034] running on GCE node
I20260620 14:30:01.512969 16855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:01.513319 16855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:01.514460 16855 hybrid_clock.cc:648] HybridClock initialized: now 1781965801514428 us; error 46 us; skew 500 ppm
Jun 20 14:30:01 dist-test-slave-hfc3 krb5kdc[16842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:01.517210 16855 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:01.517728 16855 webserver.cc:460] Webserver started at http://127.0.98.62:42491/ using document root <none> and password file <none>
I20260620 14:30:01.517935 16855 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:01.517979 16855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:01.518132 16855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:01.519263 16855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "7a78e9ed7c78471d98ac2e324855d8f3"
format_stamp: "Formatted at 2026-06-20 14:30:01 on dist-test-slave-hfc3"
server_key: "cc09a05d44cacf11b7b0a88e800903eb"
server_key_iv: "e50a14c9a2df10f2c6d0d4664cb3ae1b"
server_key_version: "encryptionkey@0"
I20260620 14:30:01.519590 16855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a78e9ed7c78471d98ac2e324855d8f3"
format_stamp: "Formatted at 2026-06-20 14:30:01 on dist-test-slave-hfc3"
server_key: "cc09a05d44cacf11b7b0a88e800903eb"
server_key_iv: "e50a14c9a2df10f2c6d0d4664cb3ae1b"
server_key_version: "encryptionkey@0"
I20260620 14:30:01.520969 16855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:30:01.521983 16870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:01.522169 16855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:01.522259 16855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "7a78e9ed7c78471d98ac2e324855d8f3"
format_stamp: "Formatted at 2026-06-20 14:30:01 on dist-test-slave-hfc3"
server_key: "cc09a05d44cacf11b7b0a88e800903eb"
server_key_iv: "e50a14c9a2df10f2c6d0d4664cb3ae1b"
server_key_version: "encryptionkey@0"
I20260620 14:30:01.522397 16855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:01.537424 16855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:01.540078 16855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:01.540205 16855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:01.544450 16855 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39051
I20260620 14:30:01.544514 16922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39051 every 8 connection(s)
I20260620 14:30:01.544902 16855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:01.545986 16923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:01.548556 16923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3: Bootstrap starting.
I20260620 14:30:01.549193 16923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:01.549497 16923 log.cc:824] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:01.550290 16923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3: No bootstrap required, opened a new log
I20260620 14:30:01.551975 16923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a78e9ed7c78471d98ac2e324855d8f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39051 } }
I20260620 14:30:01.552155 16923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:01.552205 16923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a78e9ed7c78471d98ac2e324855d8f3, State: Initialized, Role: FOLLOWER
I20260620 14:30:01.552386 16923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a78e9ed7c78471d98ac2e324855d8f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39051 } }
I20260620 14:30:01.552498 16923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:01.552544 16923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:01.552597 16923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:01.553189 16923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a78e9ed7c78471d98ac2e324855d8f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39051 } }
I20260620 14:30:01.553332 16923 leader_election.cc:302] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a78e9ed7c78471d98ac2e324855d8f3; no voters:
I20260620 14:30:01.553503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 16855
I20260620 14:30:01.553560 16923 leader_election.cc:288] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:01.553582 16926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:01.553644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:01.553700 16926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [term 1 LEADER]: Becoming Leader. State: Replica: 7a78e9ed7c78471d98ac2e324855d8f3, State: Running, Role: LEADER
I20260620 14:30:01.553774 16926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a78e9ed7c78471d98ac2e324855d8f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39051 } }
I20260620 14:30:01.553838 16923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:01.553886 392 external_mini_cluster.cc:1428] Setting key e6238a776ee0e53b9d9a82a4aa2329c1
I20260620 14:30:01.554109 16926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a78e9ed7c78471d98ac2e324855d8f3. Latest consensus state: current_term: 1 leader_uuid: "7a78e9ed7c78471d98ac2e324855d8f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a78e9ed7c78471d98ac2e324855d8f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39051 } } }
I20260620 14:30:01.554282 16926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:01.554311 16927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a78e9ed7c78471d98ac2e324855d8f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a78e9ed7c78471d98ac2e324855d8f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39051 } } }
I20260620 14:30:01.554505 16927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:01.555321 16937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:01.556094 16937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:01.557729 16937 catalog_manager.cc:1347] Generated new cluster ID: 4237a09dd731431bab084042f1e6af59
I20260620 14:30:01.557803 16937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:30:01 dist-test-slave-hfc3 krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:01.566767 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:01.554655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.0.98.62:39051)
0620 14:30:01.554843 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:01.554854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:01.555133 (+ 279us) server_negotiation.cc:407] Connection header received
0620 14:30:01.555284 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:01.555292 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:01.555391 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:01.555644 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:01.556534 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:01.557890 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:01.559081 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:01.559729 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:01.559860 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:01.562704 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:01.562709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:01.562709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:01.562723 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:01.564824 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:01.565269 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:01.565272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:01.565273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:01.565401 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:01.565650 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:01.565651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:01.565651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:01.565981 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:01.566135 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:01.566248 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 14:30:01.566416 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260620 14:30:01.575280 16937 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:01.575892 16937 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:01.579348 16937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3: Generated new TSK 0
I20260620 14:30:01.579571 16937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:30:01.609838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39051
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:01.732384 16949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:01.732595 16949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:01.732620 16949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:01.733839 16949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:01.733884 16949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:01.733912 16949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:30:01.735481 16949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:01.735736 16949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:01.735996 16949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:01.741127 16954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:01.741111 16955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:01.741111 16957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:01.741442 16949 server_base.cc:1034] running on GCE node
I20260620 14:30:01.741662 16949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:01.741866 16949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:01.743010 16949 hybrid_clock.cc:648] HybridClock initialized: now 1781965801742997 us; error 29 us; skew 500 ppm
Jun 20 14:30:01 dist-test-slave-hfc3 krb5kdc[16842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:01.745545 16949 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:30:01.746008 16949 webserver.cc:460] Webserver started at http://127.0.98.1:33355/ using document root <none> and password file <none>
I20260620 14:30:01.746243 16949 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:01.746290 16949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:01.746465 16949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:01.747540 16949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "e747232f09e44b9c922557f69a255661"
format_stamp: "Formatted at 2026-06-20 14:30:01 on dist-test-slave-hfc3"
server_key: "acc8627c19e50d1db3893c43df36671f"
server_key_iv: "7682bd763d720d8c74e5b3d95e2f0e7f"
server_key_version: "encryptionkey@0"
I20260620 14:30:01.747916 16949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e747232f09e44b9c922557f69a255661"
format_stamp: "Formatted at 2026-06-20 14:30:01 on dist-test-slave-hfc3"
server_key: "acc8627c19e50d1db3893c43df36671f"
server_key_iv: "7682bd763d720d8c74e5b3d95e2f0e7f"
server_key_version: "encryptionkey@0"
I20260620 14:30:01.749256 16949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:30:01.750084 16964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:01.750264 16949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:30:01.750388 16949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "e747232f09e44b9c922557f69a255661"
format_stamp: "Formatted at 2026-06-20 14:30:01 on dist-test-slave-hfc3"
server_key: "acc8627c19e50d1db3893c43df36671f"
server_key_iv: "7682bd763d720d8c74e5b3d95e2f0e7f"
server_key_version: "encryptionkey@0"
I20260620 14:30:01.750478 16949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:01.763813 16949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:01.765780 16949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:01.765980 16949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:01.766229 16949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:01.766600 16949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:01.766656 16949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:01.766696 16949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:01.766724 16949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:01.772609 16949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41207
I20260620 14:30:01.772684 17077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41207 every 8 connection(s)
I20260620 14:30:01.773025 16949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:30:01 dist-test-slave-hfc3 krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:01.780164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 16949
I20260620 14:30:01.780287 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:01.780509 392 external_mini_cluster.cc:1428] Setting key 86e2485633cf273799a31669f51c4d35
I20260620 14:30:01.783349 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:01.773819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60845 (local address 127.0.98.62:39051)
0620 14:30:01.773930 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:01.773933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:01.774117 (+ 184us) server_negotiation.cc:407] Connection header received
0620 14:30:01.775124 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:01.775126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:01.775138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:01.775198 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:01.775867 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:01.776382 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:01.777508 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:01.777721 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:01.777823 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:01.780012 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:01.780013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:01.780014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:01.780021 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:01.781738 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:01.782236 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:01.782237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:01.782238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:01.782291 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:01.782597 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:01.782597 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:01.782597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:01.782705 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:01.782758 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:01.783136 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 14:30:01.783238 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:30:01.783365 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:01.773727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39051 (local address 127.0.98.1:60845)
0620 14:30:01.773976 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:01.774024 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:01.775006 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:01.775269 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:01.775273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:01.775550 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:01.775746 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:01.775750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:01.776479 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:01.776480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:01.777406 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:01.777409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:01.777841 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:01.777842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:01.777965 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:01.778551 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:01.778559 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:01.779915 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:01.781822 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:01.781830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:01.781832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:01.782156 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:01.782382 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:01.782383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:01.782384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:01.782530 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:01.782772 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:01.782778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:01.782960 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 14:30:01.783074 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 14:30:01.783198 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
I20260620 14:30:01.783937 17078 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39051
I20260620 14:30:01.784024 17078 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:01.784216 17078 heartbeater.cc:511] Master 127.0.98.62:39051 requested a full tablet report, sending...
I20260620 14:30:01.784718 16887 ts_manager.cc:194] Registered new tserver with Master: e747232f09e44b9c922557f69a255661 (127.0.98.1:41207)
I20260620 14:30:01.785338 16887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60845
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:30:01.807397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39051
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:01.889616 17083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:01.889874 17083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:01.889915 17083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:01.891137 17083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:01.891211 17083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:01.891255 17083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:30:01.892762 17083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:01.893050 17083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:01.893293 17083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:01.898187 17089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:01.898187 17091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:01.898228 17088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:01.898425 17083 server_base.cc:1034] running on GCE node
I20260620 14:30:01.898646 17083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:01.898895 17083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:01.900058 17083 hybrid_clock.cc:648] HybridClock initialized: now 1781965801900038 us; error 50 us; skew 500 ppm
Jun 20 14:30:01 dist-test-slave-hfc3 krb5kdc[16842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:01.902633 17083 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:30:01.903136 17083 webserver.cc:460] Webserver started at http://127.0.98.2:33515/ using document root <none> and password file <none>
I20260620 14:30:01.903376 17083 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:01.903447 17083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:01.903595 17083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:01.904645 17083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "dfe2be2d129746a6b063296a8c9d7f9a"
format_stamp: "Formatted at 2026-06-20 14:30:01 on dist-test-slave-hfc3"
server_key: "b96478432cb25de916dffeb96efe852c"
server_key_iv: "7eed7c2be52a8599fd092615d0e9b8e4"
server_key_version: "encryptionkey@0"
I20260620 14:30:01.905027 17083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dfe2be2d129746a6b063296a8c9d7f9a"
format_stamp: "Formatted at 2026-06-20 14:30:01 on dist-test-slave-hfc3"
server_key: "b96478432cb25de916dffeb96efe852c"
server_key_iv: "7eed7c2be52a8599fd092615d0e9b8e4"
server_key_version: "encryptionkey@0"
I20260620 14:30:01.906405 17083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:01.907231 17098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:01.907444 17083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:01.907545 17083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "dfe2be2d129746a6b063296a8c9d7f9a"
format_stamp: "Formatted at 2026-06-20 14:30:01 on dist-test-slave-hfc3"
server_key: "b96478432cb25de916dffeb96efe852c"
server_key_iv: "7eed7c2be52a8599fd092615d0e9b8e4"
server_key_version: "encryptionkey@0"
I20260620 14:30:01.907626 17083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:01.916347 17083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:01.918196 17083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:01.918413 17083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:01.918639 17083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:01.918954 17083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:01.919006 17083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:01.919047 17083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:01.919076 17083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:01.925215 17083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46177
I20260620 14:30:01.925246 17211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46177 every 8 connection(s)
I20260620 14:30:01.925626 17083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:30:01 dist-test-slave-hfc3 krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:01.932507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 17083
I20260620 14:30:01.932615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:01.932854 392 external_mini_cluster.cc:1428] Setting key 934e5269069877c33cf5d49344d4af06
I20260620 14:30:01.935801 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:01.926408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56109 (local address 127.0.98.62:39051)
0620 14:30:01.926459 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:01.926461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:01.926706 (+ 245us) server_negotiation.cc:407] Connection header received
0620 14:30:01.927702 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:01.927704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:01.927717 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:01.927772 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:01.928564 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:01.929092 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:01.930149 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:01.930423 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:01.930509 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:01.932835 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:01.932837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:01.932837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:01.932845 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:01.934434 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:01.934842 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:01.934844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:01.934845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:01.934907 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:01.935157 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:01.935158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:01.935158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:01.935279 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:01.935354 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:01.935622 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:30:01.935710 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:30:01.936020 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:01.926423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39051 (local address 127.0.98.2:56109)
0620 14:30:01.926578 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:01.926614 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:01.927580 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:01.927906 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:01.927911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:01.928192 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:01.928429 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:01.928435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:01.929197 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:01.929199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:01.930037 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:01.930039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:01.930525 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:01.930526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:01.930652 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:01.931235 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:01.931244 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:01.932747 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:01.934510 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:01.934517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:01.934519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:01.934744 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:01.934972 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:01.934973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:01.934973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:01.935087 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:01.935345 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:01.935351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:01.935493 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 14:30:01.935600 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:30:01.935754 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260620 14:30:01.936578 17212 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39051
I20260620 14:30:01.936686 17212 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:01.936887 17212 heartbeater.cc:511] Master 127.0.98.62:39051 requested a full tablet report, sending...
I20260620 14:30:01.937268 16887 ts_manager.cc:194] Registered new tserver with Master: dfe2be2d129746a6b063296a8c9d7f9a (127.0.98.2:46177)
I20260620 14:30:01.937839 16887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56109
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:30:01.959393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39051
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:02.043332 17217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:02.043551 17217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:02.043576 17217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:02.044711 17217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:02.044749 17217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:02.044775 17217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:30:02.046236 17217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:02.046511 17217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:02.046716 17217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:02.051855 17223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:02.051941 17222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:02.051968 17225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:02.052275 17217 server_base.cc:1034] running on GCE node
I20260620 14:30:02.052551 17217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:02.052764 17217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:02.053910 17217 hybrid_clock.cc:648] HybridClock initialized: now 1781965802053887 us; error 37 us; skew 500 ppm
Jun 20 14:30:02 dist-test-slave-hfc3 krb5kdc[16842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:02.056854 17217 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:30:02.057335 17217 webserver.cc:460] Webserver started at http://127.0.98.3:35321/ using document root <none> and password file <none>
I20260620 14:30:02.057596 17217 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:02.057668 17217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:02.057825 17217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:02.058950 17217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "c7ddefb1f775495982153a453b35f95f"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "c1d3b875a7139a2c0d9fb17455a0c378"
server_key_iv: "bd9020080f46bd1daf6ae195202e86ad"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.059310 17217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c7ddefb1f775495982153a453b35f95f"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "c1d3b875a7139a2c0d9fb17455a0c378"
server_key_iv: "bd9020080f46bd1daf6ae195202e86ad"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.060633 17217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 14:30:02.061367 17232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.061532 17217 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:02.061640 17217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "c7ddefb1f775495982153a453b35f95f"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "c1d3b875a7139a2c0d9fb17455a0c378"
server_key_iv: "bd9020080f46bd1daf6ae195202e86ad"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.061738 17217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:02.085642 17217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:02.087774 17217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:02.087963 17217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:02.088213 17217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:02.088544 17217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:02.088603 17217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.088651 17217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:02.088675 17217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.094775 17217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33911
I20260620 14:30:02.094842 17345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33911 every 8 connection(s)
I20260620 14:30:02.095212 17217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:30:02 dist-test-slave-hfc3 krb5kdc[16842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:02.104987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 17217
I20260620 14:30:02.105093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:02.105212 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.096060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42057 (local address 127.0.98.62:39051)
0620 14:30:02.096137 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.096139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.096387 (+ 248us) server_negotiation.cc:407] Connection header received
0620 14:30:02.097310 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.097312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.097324 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.097391 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:02.098000 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.098595 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.099752 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.100000 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.100061 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.102299 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:02.102300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:02.102301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:02.102307 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:02.103933 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:02.104319 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:02.104321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:02.104321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:02.104378 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:02.104596 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:02.104596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:02.104597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:02.104701 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:02.104780 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.105026 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.105114 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:30:02.105296 392 external_mini_cluster.cc:1428] Setting key ebf9925f8d39b00627b59b5e7f8ae952
I20260620 14:30:02.105371 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.096048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39051 (local address 127.0.98.3:42057)
0620 14:30:02.096231 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:02.096270 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:02.097202 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:02.097452 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:02.097455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:02.097715 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:02.097910 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.097914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.098704 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.098714 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.099592 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.099595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.100079 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.100080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.100227 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.100734 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:02.100742 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:02.102146 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:02.104000 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:02.104005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:02.104007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:02.104212 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:02.104444 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:02.104444 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:02.104444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:02.104539 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:02.104756 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:02.104761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:02.104890 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 14:30:02.104975 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:30:02.105135 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
I20260620 14:30:02.105942 17346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39051
I20260620 14:30:02.106045 17346 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:02.106231 17346 heartbeater.cc:511] Master 127.0.98.62:39051 requested a full tablet report, sending...
I20260620 14:30:02.106621 16887 ts_manager.cc:194] Registered new tserver with Master: c7ddefb1f775495982153a453b35f95f (127.0.98.3:33911)
I20260620 14:30:02.106963 16887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42057
I20260620 14:30:02.116961 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:30:02.125032 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.118357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60288 (local address 127.0.98.62:39051)
0620 14:30:02.118429 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.118432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.118523 (+ 91us) server_negotiation.cc:407] Connection header received
0620 14:30:02.118604 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.118604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.118616 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.118661 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:02.119224 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.119717 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.120756 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.120952 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.121032 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.122320 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:02.122322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:02.122322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:02.122330 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:02.123822 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:02.124202 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:02.124204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:02.124204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:02.124270 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:02.124476 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:02.124477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:02.124477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:02.124634 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:02.124743 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.124847 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.124929 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:30:02.126679 16887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:02.128374 16887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:02.135072 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.130733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46177 (local address 127.0.0.1:56942)
0620 14:30:02.131074 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:02.131118 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:02.131191 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:02.131655 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:02.131656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:02.131658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:02.131727 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.131729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.132667 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.132669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.133878 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.133881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.134757 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.134759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.134855 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.134871 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:30:02.134945 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:30:02.134994 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":5,"thread_start_us":80,"threads_started":1}
I20260620 14:30:02.135113 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.130859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.0.98.2:46177)
0620 14:30:02.131142 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.131147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.131167 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:30:02.131525 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.131528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.131545 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.131808 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:02.131899 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.132566 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.133962 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.134671 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.134750 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.134818 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.134979 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.135031 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
I20260620 14:30:02.135264 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.130733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33911 (local address 127.0.0.1:49504)
0620 14:30:02.131289 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:02.131298 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:02.131354 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:02.131641 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:02.131644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:02.131646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:02.131737 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.131739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.133094 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.133095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.134107 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.134109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.135031 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.135032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.135105 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.135110 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:30:02.135143 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 14:30:02.135187 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":7,"thread_start_us":82,"threads_started":1}
I20260620 14:30:02.136044 17146 tablet_service.cc:1467] Processing CreateTablet for tablet 5523d642ef5844a984fbb60d2998c072 (DEFAULT_TABLE table=test-table [id=acf2b528dd5f443a8b8fb742e80d328d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:02.136440 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.130982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41207 (local address 127.0.0.1:45036)
0620 14:30:02.131515 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:02.131524 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:02.131569 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:02.131763 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:02.131764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:02.131766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:02.131807 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.131809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.134177 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.134178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.135096 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.135099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.136202 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.136203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.136266 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.136270 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:30:02.136327 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:30:02.136367 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260620 14:30:02.136443 17146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5523d642ef5844a984fbb60d2998c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:02.136596 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.130859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.0.98.3:33911)
0620 14:30:02.131265 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.131269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.131454 (+ 185us) server_negotiation.cc:407] Connection header received
0620 14:30:02.131505 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.131508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.131527 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.131826 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:02.131906 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.132979 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.134183 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.134909 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.136294 (+ 1385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.136366 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.136396 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.136429 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":109,"threads_started":1}
I20260620 14:30:02.136662 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.131121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.0.98.1:41207)
0620 14:30:02.131329 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.131332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.131626 (+ 294us) server_negotiation.cc:407] Connection header received
0620 14:30:02.131668 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.131670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.131684 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.131754 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:02.131865 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.132980 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.135449 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.136116 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.136182 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.136221 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.136546 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.136591 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260620 14:30:02.137245 17280 tablet_service.cc:1467] Processing CreateTablet for tablet 5523d642ef5844a984fbb60d2998c072 (DEFAULT_TABLE table=test-table [id=acf2b528dd5f443a8b8fb742e80d328d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:02.137344 17012 tablet_service.cc:1467] Processing CreateTablet for tablet 5523d642ef5844a984fbb60d2998c072 (DEFAULT_TABLE table=test-table [id=acf2b528dd5f443a8b8fb742e80d328d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:02.137578 17280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5523d642ef5844a984fbb60d2998c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:02.137722 17012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5523d642ef5844a984fbb60d2998c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:02.139936 17363 tablet_bootstrap.cc:492] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a: Bootstrap starting.
I20260620 14:30:02.140587 17364 tablet_bootstrap.cc:492] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661: Bootstrap starting.
I20260620 14:30:02.140738 17363 tablet_bootstrap.cc:654] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:02.140944 17365 tablet_bootstrap.cc:492] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f: Bootstrap starting.
I20260620 14:30:02.141216 17364 tablet_bootstrap.cc:654] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:02.141378 17363 log.cc:824] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:02.141621 17365 tablet_bootstrap.cc:654] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:02.141794 17364 log.cc:824] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:02.142179 17365 log.cc:824] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:02.142279 17363 tablet_bootstrap.cc:492] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a: No bootstrap required, opened a new log
I20260620 14:30:02.142371 17363 ts_tablet_manager.cc:1397] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:02.142905 17365 tablet_bootstrap.cc:492] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f: No bootstrap required, opened a new log
I20260620 14:30:02.142972 17365 ts_tablet_manager.cc:1397] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:02.143129 17364 tablet_bootstrap.cc:492] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661: No bootstrap required, opened a new log
I20260620 14:30:02.143179 17364 ts_tablet_manager.cc:1397] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:30:02.143993 17363 raft_consensus.cc:348] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ddefb1f775495982153a453b35f95f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33911 } } peers { permanent_uuid: "e747232f09e44b9c922557f69a255661" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41207 } } peers { permanent_uuid: "dfe2be2d129746a6b063296a8c9d7f9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46177 } }
I20260620 14:30:02.144122 17363 raft_consensus.cc:374] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:02.144142 17363 raft_consensus.cc:729] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe2be2d129746a6b063296a8c9d7f9a, State: Initialized, Role: FOLLOWER
I20260620 14:30:02.144259 17363 consensus_queue.cc:260] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ddefb1f775495982153a453b35f95f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33911 } } peers { permanent_uuid: "e747232f09e44b9c922557f69a255661" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41207 } } peers { permanent_uuid: "dfe2be2d129746a6b063296a8c9d7f9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46177 } }
I20260620 14:30:02.144521 17212 heartbeater.cc:503] Master 127.0.98.62:39051 was elected leader, sending a full tablet report...
I20260620 14:30:02.144522 17363 ts_tablet_manager.cc:1428] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:02.144881 17365 raft_consensus.cc:348] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ddefb1f775495982153a453b35f95f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33911 } } peers { permanent_uuid: "e747232f09e44b9c922557f69a255661" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41207 } } peers { permanent_uuid: "dfe2be2d129746a6b063296a8c9d7f9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46177 } }
I20260620 14:30:02.145040 17365 raft_consensus.cc:374] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:02.145076 17365 raft_consensus.cc:729] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ddefb1f775495982153a453b35f95f, State: Initialized, Role: FOLLOWER
I20260620 14:30:02.145186 17365 consensus_queue.cc:260] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ddefb1f775495982153a453b35f95f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33911 } } peers { permanent_uuid: "e747232f09e44b9c922557f69a255661" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41207 } } peers { permanent_uuid: "dfe2be2d129746a6b063296a8c9d7f9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46177 } }
I20260620 14:30:02.145130 17364 raft_consensus.cc:348] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ddefb1f775495982153a453b35f95f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33911 } } peers { permanent_uuid: "e747232f09e44b9c922557f69a255661" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41207 } } peers { permanent_uuid: "dfe2be2d129746a6b063296a8c9d7f9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46177 } }
I20260620 14:30:02.145272 17364 raft_consensus.cc:374] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:02.145329 17364 raft_consensus.cc:729] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e747232f09e44b9c922557f69a255661, State: Initialized, Role: FOLLOWER
I20260620 14:30:02.145463 17365 ts_tablet_manager.cc:1428] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:02.145541 17346 heartbeater.cc:503] Master 127.0.98.62:39051 was elected leader, sending a full tablet report...
I20260620 14:30:02.145438 17364 consensus_queue.cc:260] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ddefb1f775495982153a453b35f95f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33911 } } peers { permanent_uuid: "e747232f09e44b9c922557f69a255661" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41207 } } peers { permanent_uuid: "dfe2be2d129746a6b063296a8c9d7f9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46177 } }
I20260620 14:30:02.146005 17364 ts_tablet_manager.cc:1428] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:02.146021 17078 heartbeater.cc:503] Master 127.0.98.62:39051 was elected leader, sending a full tablet report...
W20260620 14:30:02.176272 17213 tablet.cc:2369] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:30:02.273790 17079 tablet.cc:2369] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:30:02.345178 17370 raft_consensus.cc:482] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:02.345304 17370 raft_consensus.cc:504] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ddefb1f775495982153a453b35f95f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33911 } } peers { permanent_uuid: "e747232f09e44b9c922557f69a255661" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41207 } } peers { permanent_uuid: "dfe2be2d129746a6b063296a8c9d7f9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46177 } }
I20260620 14:30:02.345656 17370 leader_election.cc:288] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e747232f09e44b9c922557f69a255661 (127.0.98.1:41207), dfe2be2d129746a6b063296a8c9d7f9a (127.0.98.2:46177)
W20260620 14:30:02.345801 17347 tablet.cc:2369] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:30:02.348719 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.346082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53301 (local address 127.0.98.1:41207)
0620 14:30:02.346153 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.346155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.346165 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:30:02.346202 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.346203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.346212 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.346281 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:02.346500 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.347004 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.348005 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.348366 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.348447 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.348458 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.348601 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.348639 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:30:02.348757 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.345987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46999 (local address 127.0.98.2:46177)
0620 14:30:02.346035 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.346037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.346180 (+ 143us) server_negotiation.cc:407] Connection header received
0620 14:30:02.346219 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.346220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.346234 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.346371 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:02.346539 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.347043 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.348013 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.348395 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.348478 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.348487 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.348647 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.348686 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:30:02.348770 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.345871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41207 (local address 127.0.98.3:53301)
0620 14:30:02.346034 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:02.346046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:02.346120 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:02.346285 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:02.346286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:02.346287 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:02.346412 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.346415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.347095 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.347096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.347910 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.347913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.348444 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.348445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.348527 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.348530 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:02.348596 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:30:02.348635 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":41}
I20260620 14:30:02.348970 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.345872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46177 (local address 127.0.98.3:46999)
0620 14:30:02.346084 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:02.346096 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:02.346167 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:02.346377 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:02.346378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:02.346380 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:02.346458 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.346459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.347128 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.347129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.347918 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.347920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.348478 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.348479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.348561 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.348564 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:02.348638 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:30:02.348676 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":11,"thread_start_us":97,"threads_started":1}
I20260620 14:30:02.349124 17032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5523d642ef5844a984fbb60d2998c072" candidate_uuid: "c7ddefb1f775495982153a453b35f95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e747232f09e44b9c922557f69a255661" is_pre_election: true
I20260620 14:30:02.349366 17032 raft_consensus.cc:2454] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7ddefb1f775495982153a453b35f95f in term 0.
I20260620 14:30:02.349334 17166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5523d642ef5844a984fbb60d2998c072" candidate_uuid: "c7ddefb1f775495982153a453b35f95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfe2be2d129746a6b063296a8c9d7f9a" is_pre_election: true
I20260620 14:30:02.349462 17166 raft_consensus.cc:2454] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7ddefb1f775495982153a453b35f95f in term 0.
I20260620 14:30:02.349592 17234 leader_election.cc:302] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7ddefb1f775495982153a453b35f95f, e747232f09e44b9c922557f69a255661; no voters:
I20260620 14:30:02.349716 17370 raft_consensus.cc:2781] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:02.349771 17370 raft_consensus.cc:482] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:02.349830 17370 raft_consensus.cc:3037] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:02.350636 17370 raft_consensus.cc:504] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ddefb1f775495982153a453b35f95f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33911 } } peers { permanent_uuid: "e747232f09e44b9c922557f69a255661" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41207 } } peers { permanent_uuid: "dfe2be2d129746a6b063296a8c9d7f9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46177 } }
I20260620 14:30:02.350793 17370 leader_election.cc:288] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [CANDIDATE]: Term 1 election: Requested vote from peers e747232f09e44b9c922557f69a255661 (127.0.98.1:41207), dfe2be2d129746a6b063296a8c9d7f9a (127.0.98.2:46177)
I20260620 14:30:02.350970 17032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5523d642ef5844a984fbb60d2998c072" candidate_uuid: "c7ddefb1f775495982153a453b35f95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e747232f09e44b9c922557f69a255661"
I20260620 14:30:02.350989 17166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5523d642ef5844a984fbb60d2998c072" candidate_uuid: "c7ddefb1f775495982153a453b35f95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfe2be2d129746a6b063296a8c9d7f9a"
I20260620 14:30:02.351055 17166 raft_consensus.cc:3037] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:02.351058 17032 raft_consensus.cc:3037] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:02.351749 17032 raft_consensus.cc:2454] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7ddefb1f775495982153a453b35f95f in term 1.
I20260620 14:30:02.351810 17166 raft_consensus.cc:2454] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7ddefb1f775495982153a453b35f95f in term 1.
I20260620 14:30:02.351961 17234 leader_election.cc:302] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7ddefb1f775495982153a453b35f95f, e747232f09e44b9c922557f69a255661; no voters:
I20260620 14:30:02.352099 17370 raft_consensus.cc:2781] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:02.352303 17370 raft_consensus.cc:686] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 1 LEADER]: Becoming Leader. State: Replica: c7ddefb1f775495982153a453b35f95f, State: Running, Role: LEADER
I20260620 14:30:02.352413 17370 consensus_queue.cc:237] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ddefb1f775495982153a453b35f95f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33911 } } peers { permanent_uuid: "e747232f09e44b9c922557f69a255661" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41207 } } peers { permanent_uuid: "dfe2be2d129746a6b063296a8c9d7f9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46177 } }
I20260620 14:30:02.353277 16887 catalog_manager.cc:5495] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f reported cstate change: term changed from 0 to 1, leader changed from <none> to c7ddefb1f775495982153a453b35f95f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c7ddefb1f775495982153a453b35f95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ddefb1f775495982153a453b35f95f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e747232f09e44b9c922557f69a255661" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfe2be2d129746a6b063296a8c9d7f9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46177 } health_report { overall_health: UNKNOWN } } }
I20260620 14:30:02.360795 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.358598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.0.98.3:33911)
0620 14:30:02.358664 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.358666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.358695 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:30:02.358731 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.358732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.358747 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.358805 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:02.358920 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.359367 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.360089 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.360268 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.360320 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.360455 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:02.360551 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:02.360609 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.360641 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.360658 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:30:02.362592 17032 raft_consensus.cc:1261] T 5523d642ef5844a984fbb60d2998c072 P e747232f09e44b9c922557f69a255661 [term 1 FOLLOWER]: Refusing update from remote peer c7ddefb1f775495982153a453b35f95f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:02.362603 17166 raft_consensus.cc:1261] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a [term 1 FOLLOWER]: Refusing update from remote peer c7ddefb1f775495982153a453b35f95f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:02.362995 17370 consensus_queue.cc:1035] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfe2be2d129746a6b063296a8c9d7f9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:02.363128 17373 consensus_queue.cc:1035] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e747232f09e44b9c922557f69a255661" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:02.365875 17377 mvcc.cc:205] Tried to move back new op lower bound from 7298931926474088448 to 7298931926436081664. Current Snapshot: MvccSnapshot[applied={T|T < 7298931926474088448}]
I20260620 14:30:02.366433 17378 mvcc.cc:205] Tried to move back new op lower bound from 7298931926474088448 to 7298931926436081664. Current Snapshot: MvccSnapshot[applied={T|T < 7298931926474088448}]
I20260620 14:30:02.369026 16886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:02.369131 16886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:02.370047 16886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a78e9ed7c78471d98ac2e324855d8f3: Sending DeleteTablet for 3 replicas of tablet 5523d642ef5844a984fbb60d2998c072
I20260620 14:30:02.370493 17012 tablet_service.cc:1514] Processing DeleteTablet for tablet 5523d642ef5844a984fbb60d2998c072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45036
I20260620 14:30:02.370497 17280 tablet_service.cc:1514] Processing DeleteTablet for tablet 5523d642ef5844a984fbb60d2998c072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49504
I20260620 14:30:02.370746 17146 tablet_service.cc:1514] Processing DeleteTablet for tablet 5523d642ef5844a984fbb60d2998c072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56942
I20260620 14:30:02.370765 17387 tablet_replica.cc:330] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f: stopping tablet replica
I20260620 14:30:02.370872 17387 raft_consensus.cc:2229] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:30:02.371006 17387 raft_consensus.cc:2258] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:02.371007 17388 tablet_replica.cc:330] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a: stopping tablet replica
I20260620 14:30:02.371201 17388 raft_consensus.cc:2229] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:02.371245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 16949
I20260620 14:30:02.371469 17387 ts_tablet_manager.cc:1905] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:02.371846 17388 raft_consensus.cc:2258] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:02.372973 17388 ts_tablet_manager.cc:1905] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:02.373184 17387 ts_tablet_manager.cc:1918] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:02.373266 17387 log.cc:1196] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/5523d642ef5844a984fbb60d2998c072
I20260620 14:30:02.373549 17387 ts_tablet_manager.cc:1939] T 5523d642ef5844a984fbb60d2998c072 P c7ddefb1f775495982153a453b35f95f: Deleting consensus metadata
I20260620 14:30:02.375308 17388 ts_tablet_manager.cc:1918] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:02.375454 17388 log.cc:1196] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/5523d642ef5844a984fbb60d2998c072
I20260620 14:30:02.375798 17388 ts_tablet_manager.cc:1939] T 5523d642ef5844a984fbb60d2998c072 P dfe2be2d129746a6b063296a8c9d7f9a: Deleting consensus metadata
I20260620 14:30:02.376271 16873 catalog_manager.cc:4841] TS dfe2be2d129746a6b063296a8c9d7f9a (127.0.98.2:46177): tablet 5523d642ef5844a984fbb60d2998c072 (table test-table [id=acf2b528dd5f443a8b8fb742e80d328d]) successfully deleted
I20260620 14:30:02.376605 16872 catalog_manager.cc:4841] TS c7ddefb1f775495982153a453b35f95f (127.0.98.3:33911): tablet 5523d642ef5844a984fbb60d2998c072 (table test-table [id=acf2b528dd5f443a8b8fb742e80d328d]) successfully deleted
W20260620 14:30:02.379941 16872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41207 (error 108)
I20260620 14:30:02.380558 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.380414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41207 (local address 127.0.0.1:45038)
0620 14:30:02.380478 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:02.380523 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:30:02.380575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 17083
W20260620 14:30:02.382476 16872 catalog_manager.cc:4584] TS e747232f09e44b9c922557f69a255661 (127.0.98.1:41207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5523d642ef5844a984fbb60d2998c072: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41207: connect: Connection refused (error 111)
I20260620 14:30:02.385673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 17217
I20260620 14:30:02.390147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 16855
2026-06-20T14:30:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2144 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T14:30:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:02Z Disabled control of system clock
I20260620 14:30:02.418572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46055
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:30:02.500478 17393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:02.500787 17393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:02.500847 17393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:02.502043 17393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:02.502084 17393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:02.502096 17393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:02.502110 17393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:02.502122 17393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:02.503577 17393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:02.503789 17393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:02.504057 17393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:02.509029 17399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:02.509140 17398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:02.509034 17401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:02.509289 17393 server_base.cc:1034] running on GCE node
I20260620 14:30:02.509518 17393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:02.509786 17393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:02.510957 17393 hybrid_clock.cc:648] HybridClock initialized: now 1781965802510940 us; error 36 us; skew 500 ppm
I20260620 14:30:02.512382 17393 webserver.cc:460] Webserver started at http://127.0.98.62:34339/ using document root <none> and password file <none>
I20260620 14:30:02.512662 17393 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:02.512748 17393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:02.512904 17393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:02.514050 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "384c20c9ea7e45f1946849e352a29073"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "c2589fb44e883702d7a05aa523b214d9"
server_key_iv: "72568354a7828def966cabb87291ce39"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.514492 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "384c20c9ea7e45f1946849e352a29073"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "c2589fb44e883702d7a05aa523b214d9"
server_key_iv: "72568354a7828def966cabb87291ce39"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.515854 17393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:30:02.516610 17407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.516782 17393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:30:02.516885 17393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "384c20c9ea7e45f1946849e352a29073"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "c2589fb44e883702d7a05aa523b214d9"
server_key_iv: "72568354a7828def966cabb87291ce39"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.516966 17393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:02.537856 17393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:02.538226 17393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:02.538432 17393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:02.542626 17393 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46055
I20260620 14:30:02.542677 17459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46055 every 8 connection(s)
I20260620 14:30:02.543081 17393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:02.543442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 17393
I20260620 14:30:02.543586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:02.543776 392 external_mini_cluster.cc:1428] Setting key e872b59e64a21d28fd8a708f09983ef3
I20260620 14:30:02.544354 17460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:02.547667 17460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073: Bootstrap starting.
I20260620 14:30:02.548292 17460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:02.548664 17460 log.cc:824] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:02.549475 17460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073: No bootstrap required, opened a new log
I20260620 14:30:02.549989 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.544385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.0.98.62:46055)
0620 14:30:02.544585 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.544595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.544628 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:30:02.545161 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.545170 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.545256 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.545414 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:02.545586 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.546997 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.547837 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.548260 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.548413 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.549259 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:02.549262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:02.549263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:02.549434 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:02.549468 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:02.549479 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:02.549560 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.549668 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.549755 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260620 14:30:02.551611 17460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384c20c9ea7e45f1946849e352a29073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } }
I20260620 14:30:02.551756 17460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:02.551781 17460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384c20c9ea7e45f1946849e352a29073, State: Initialized, Role: FOLLOWER
I20260620 14:30:02.551895 17460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384c20c9ea7e45f1946849e352a29073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } }
I20260620 14:30:02.551986 17460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:02.552009 17460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:02.552038 17460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:02.552599 17460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384c20c9ea7e45f1946849e352a29073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } }
I20260620 14:30:02.552690 17460 leader_election.cc:302] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384c20c9ea7e45f1946849e352a29073; no voters:
I20260620 14:30:02.552837 17460 leader_election.cc:288] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:02.552906 17465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:02.553043 17465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [term 1 LEADER]: Becoming Leader. State: Replica: 384c20c9ea7e45f1946849e352a29073, State: Running, Role: LEADER
I20260620 14:30:02.553118 17460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:02.553153 17465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384c20c9ea7e45f1946849e352a29073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } }
I20260620 14:30:02.553501 17467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "384c20c9ea7e45f1946849e352a29073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384c20c9ea7e45f1946849e352a29073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } }
I20260620 14:30:02.553537 17466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384c20c9ea7e45f1946849e352a29073. Latest consensus state: current_term: 1 leader_uuid: "384c20c9ea7e45f1946849e352a29073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384c20c9ea7e45f1946849e352a29073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } }
I20260620 14:30:02.553710 17466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:02.553669 17467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:02.554201 17472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:02.554766 17472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:02.556321 17472 catalog_manager.cc:1347] Generated new cluster ID: e5b64a7e02da4dde9e02e26dc9c70204
I20260620 14:30:02.556380 17472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:02.566514 17472 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:02.567094 17472 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:02.572535 17472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073: Generated new TSK 0
I20260620 14:30:02.572769 17472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:02.581496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--builtin_ntp_servers=127.0.98.20:37387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:30:02.665441 17484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:02.665675 17484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:02.665700 17484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:02.667163 17484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:02.667224 17484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:02.667253 17484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:30:02.668720 17484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:02.668987 17484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:02.669324 17484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:02.675618 17490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:02.675815 17492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:02.675686 17484 server_base.cc:1034] running on GCE node
W20260620 14:30:02.675647 17489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:02.676144 17484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:02.676461 17484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:02.677636 17484 hybrid_clock.cc:648] HybridClock initialized: now 1781965802677624 us; error 29 us; skew 500 ppm
I20260620 14:30:02.679021 17484 webserver.cc:460] Webserver started at http://127.0.98.1:41355/ using document root <none> and password file <none>
I20260620 14:30:02.679288 17484 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:02.679374 17484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:02.679531 17484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:02.680646 17484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "00793372fd89498586b42b3fa357db42"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "9c8a7064dcabedfaf605fef527250ee4"
server_key_iv: "06f83113e91c4c1afe00999f1fb1f8a0"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.681035 17484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "00793372fd89498586b42b3fa357db42"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "9c8a7064dcabedfaf605fef527250ee4"
server_key_iv: "06f83113e91c4c1afe00999f1fb1f8a0"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.682493 17484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 14:30:02.683363 17498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.683728 17484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:02.683873 17484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "00793372fd89498586b42b3fa357db42"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "9c8a7064dcabedfaf605fef527250ee4"
server_key_iv: "06f83113e91c4c1afe00999f1fb1f8a0"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.683956 17484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:02.712512 17484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:02.712869 17484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:02.713032 17484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:02.713294 17484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:02.713634 17484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:02.713691 17484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.713755 17484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:02.713807 17484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.719828 17484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43817
I20260620 14:30:02.719928 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43817 every 8 connection(s)
I20260620 14:30:02.720248 17484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:02.725425 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.721022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38469 (local address 127.0.98.62:46055)
0620 14:30:02.721077 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.721079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.721337 (+ 258us) server_negotiation.cc:407] Connection header received
0620 14:30:02.721808 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.721809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.721823 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.721889 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:02.722624 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.723117 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.724052 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.724243 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.724318 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.725220 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:02.725221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:02.725222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:02.725225 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:02.725235 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:02.725238 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:02.725276 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.725343 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.725365 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:30:02.725662 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.721022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.1:38469)
0620 14:30:02.721206 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:02.721241 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:02.721719 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:02.721943 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:02.721946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:02.721954 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:02.722535 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.722541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.723203 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.723203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.723980 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.723982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.724342 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.724343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.724463 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.725091 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:02.725091 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:02.725121 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:02.725143 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:02.725279 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:02.725282 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:02.725284 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:30:02.725335 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:30:02.725412 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260620 14:30:02.726286 17612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46055
I20260620 14:30:02.726431 17612 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:02.726616 17612 heartbeater.cc:511] Master 127.0.98.62:46055 requested a full tablet report, sending...
I20260620 14:30:02.727087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 17484
I20260620 14:30:02.727126 17424 ts_manager.cc:194] Registered new tserver with Master: 00793372fd89498586b42b3fa357db42 (127.0.98.1:43817)
I20260620 14:30:02.727172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:02.727413 392 external_mini_cluster.cc:1428] Setting key b6a05a4ef681c7d0dc2fd4df0d0f24ce
I20260620 14:30:02.728024 17424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38469
I20260620 14:30:02.728549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--builtin_ntp_servers=127.0.98.20:37387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:30:02.811812 17615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:02.812026 17615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:02.812049 17615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:02.813231 17615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:02.813272 17615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:02.813303 17615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:30:02.814721 17615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:02.814913 17615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:02.815171 17615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:02.820065 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:02.820151 17623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:02.820080 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:02.820456 17615 server_base.cc:1034] running on GCE node
I20260620 14:30:02.820729 17615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:02.821000 17615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:02.822196 17615 hybrid_clock.cc:648] HybridClock initialized: now 1781965802822177 us; error 31 us; skew 500 ppm
I20260620 14:30:02.823503 17615 webserver.cc:460] Webserver started at http://127.0.98.2:32801/ using document root <none> and password file <none>
I20260620 14:30:02.823781 17615 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:02.823856 17615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:02.824018 17615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:02.825085 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "9e7ac5a45f84423a9b46f60ca5d2e161"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "2da616404e578f5edc7f28c1cb8d7370"
server_key_iv: "a9d897e487fd7b24c53dbce457591862"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.825481 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9e7ac5a45f84423a9b46f60ca5d2e161"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "2da616404e578f5edc7f28c1cb8d7370"
server_key_iv: "a9d897e487fd7b24c53dbce457591862"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.826864 17615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:02.827600 17629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.827759 17615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:02.827865 17615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "9e7ac5a45f84423a9b46f60ca5d2e161"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "2da616404e578f5edc7f28c1cb8d7370"
server_key_iv: "a9d897e487fd7b24c53dbce457591862"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.827940 17615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:02.850114 17615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:02.850545 17615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:02.850713 17615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:02.850976 17615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:02.851410 17615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:02.851470 17615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.851514 17615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:02.851562 17615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.857751 17615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44175
I20260620 14:30:02.857805 17742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44175 every 8 connection(s)
I20260620 14:30:02.858213 17615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:02.863646 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.859051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57335 (local address 127.0.98.62:46055)
0620 14:30:02.859128 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.859130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.859430 (+ 300us) server_negotiation.cc:407] Connection header received
0620 14:30:02.859912 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.859914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.859925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.859990 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:02.860655 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.861134 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.862094 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.862417 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.862489 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.863391 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:02.863391 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:02.863392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:02.863396 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:02.863406 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:02.863409 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:02.863447 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.863511 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.863531 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:30:02.863793 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.859054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.2:57335)
0620 14:30:02.859255 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:02.859295 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:02.859799 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:02.860056 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:02.860059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:02.860067 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:02.860573 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.860577 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.861215 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.861215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.862015 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.862017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.862512 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.862513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.862632 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.863257 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:02.863257 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:02.863291 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:02.863314 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:02.863449 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:02.863451 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:02.863454 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:02.863501 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:30:02.863580 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260620 14:30:02.863823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 17615
I20260620 14:30:02.863947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:02.864236 392 external_mini_cluster.cc:1428] Setting key 078c3c6a647da574f65502ebe1a7595a
I20260620 14:30:02.864372 17743 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46055
I20260620 14:30:02.864491 17743 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:02.864686 17743 heartbeater.cc:511] Master 127.0.98.62:46055 requested a full tablet report, sending...
I20260620 14:30:02.865041 17424 ts_manager.cc:194] Registered new tserver with Master: 9e7ac5a45f84423a9b46f60ca5d2e161 (127.0.98.2:44175)
I20260620 14:30:02.865248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--builtin_ntp_servers=127.0.98.20:37387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:30:02.865520 17424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57335
W20260620 14:30:02.946952 17746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:02.947147 17746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:02.947176 17746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:02.948330 17746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:02.948374 17746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:02.948397 17746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:30:02.949739 17746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:02.949997 17746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:02.950314 17746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:02.955729 17752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:02.955724 17751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:02.955724 17754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:02.956064 17746 server_base.cc:1034] running on GCE node
I20260620 14:30:02.956311 17746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:02.956568 17746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:02.957727 17746 hybrid_clock.cc:648] HybridClock initialized: now 1781965802957720 us; error 36 us; skew 500 ppm
I20260620 14:30:02.959091 17746 webserver.cc:460] Webserver started at http://127.0.98.3:36445/ using document root <none> and password file <none>
I20260620 14:30:02.959385 17746 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:02.959462 17746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:02.959611 17746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:02.960692 17746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "13b554428e5c4fa995c468f8476321e6"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "7bad8de509dfbdd50d389e8f37a2ac5e"
server_key_iv: "04b1707355f16e92a33c0371cc0aa221"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.961041 17746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "13b554428e5c4fa995c468f8476321e6"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "7bad8de509dfbdd50d389e8f37a2ac5e"
server_key_iv: "04b1707355f16e92a33c0371cc0aa221"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.962412 17746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:30:02.963157 17760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.963333 17746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:02.963433 17746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "13b554428e5c4fa995c468f8476321e6"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-hfc3"
server_key: "7bad8de509dfbdd50d389e8f37a2ac5e"
server_key_iv: "04b1707355f16e92a33c0371cc0aa221"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.963511 17746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:02.981554 17746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:02.981877 17746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:02.982036 17746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:02.982268 17746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:02.982630 17746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:02.982685 17746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.982720 17746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:02.982749 17746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.988446 17746 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46471
I20260620 14:30:02.988519 17873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46471 every 8 connection(s)
I20260620 14:30:02.988863 17746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:02.990495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 17746
I20260620 14:30:02.990602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:02.990803 392 external_mini_cluster.cc:1428] Setting key 5187a7cf23f597ff2712b4a51d888674
I20260620 14:30:02.994360 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.989614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37623 (local address 127.0.98.62:46055)
0620 14:30:02.989660 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.989662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.989906 (+ 244us) server_negotiation.cc:407] Connection header received
0620 14:30:02.990461 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.990463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.990474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.990525 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:02.991382 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.991907 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.992923 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.993152 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.993227 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.994050 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:02.994052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:02.994052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:02.994057 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:02.994072 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:02.994076 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:02.994124 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.994218 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.994246 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:30:02.994571 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.989592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.3:37623)
0620 14:30:02.989758 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:02.989791 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:02.990328 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:02.990619 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:02.990624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:02.990635 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:02.991291 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.991298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.992009 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.992010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.992815 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:02.992817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:02.993246 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:02.993248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:02.993377 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:02.993881 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:02.993882 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:02.993911 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:02.993933 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:02.994124 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:02.994129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:02.994132 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:02.994223 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:30:02.994328 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260620 14:30:02.994983 17874 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46055
I20260620 14:30:02.995074 17874 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:02.995246 17874 heartbeater.cc:511] Master 127.0.98.62:46055 requested a full tablet report, sending...
I20260620 14:30:02.995618 17424 ts_manager.cc:194] Registered new tserver with Master: 13b554428e5c4fa995c468f8476321e6 (127.0.98.3:46471)
I20260620 14:30:02.996006 17424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37623
I20260620 14:30:03.002521 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:30:03.006888 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.003878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36866 (local address 127.0.98.62:46055)
0620 14:30:03.003951 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:03.003953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:03.004036 (+ 83us) server_negotiation.cc:407] Connection header received
0620 14:30:03.004074 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:03.004074 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:03.004082 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:03.004129 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:03.004263 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.004734 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.005452 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.005625 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.005675 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.006641 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:03.006643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:03.006643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:03.006647 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:03.006660 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:03.006664 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:03.006710 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:03.006755 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 14:30:03.006775 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:30:03.008459 17424 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:03.010020 17424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:03.016736 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.012292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46471 (local address 127.0.0.1:57324)
0620 14:30:03.012490 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:03.012531 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:03.012595 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:03.012820 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:03.012823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:03.012826 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:03.012904 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:03.012906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.013989 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.013990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:03.015379 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:03.015382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.016421 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.016423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:03.016510 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.016523 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:30:03.016595 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:30:03.016650 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260620 14:30:03.017063 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.012397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.0.98.3:46471)
0620 14:30:03.012603 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:03.012607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:03.012620 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:30:03.012694 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:03.012696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:03.012714 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:03.012802 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:03.013002 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.013885 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.015479 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.016222 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.016294 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.016384 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:03.016935 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 14:30:03.016979 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
I20260620 14:30:03.017580 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.012324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43817 (local address 127.0.0.1:58292)
0620 14:30:03.012697 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:03.012708 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:03.012761 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:03.013207 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:03.013208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:03.013212 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:03.013267 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:03.013269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.014607 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.014609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:03.015707 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:03.015710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.017345 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.017346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:03.017431 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.017436 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:30:03.017467 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 14:30:03.017509 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":24,"thread_start_us":105,"threads_started":1}
I20260620 14:30:03.018021 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.012507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.0.98.1:43817)
0620 14:30:03.013029 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:03.013033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:03.013042 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:30:03.013080 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:03.013082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:03.013096 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:03.013341 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:03.013402 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.014508 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.015966 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.017221 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.017751 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.017849 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:03.017872 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 14:30:03.017911 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":94,"threads_started":1}
I20260620 14:30:03.018066 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.012526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44175 (local address 127.0.0.1:33666)
0620 14:30:03.012873 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:03.012881 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:03.012917 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:03.013207 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:03.013208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:03.013210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:03.013267 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:03.013268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.014652 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.014653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:03.016772 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:03.016775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.017840 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.017841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:03.017900 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.017904 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:30:03.017953 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:30:03.017990 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":28,"thread_start_us":78,"threads_started":1}
I20260620 14:30:03.017957 17808 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc36fd6556344159f0d8419e17fe5c1 (DEFAULT_TABLE table=test-table [id=5e8c29a5297b459e853b3583de08a98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:03.018397 17808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc36fd6556344159f0d8419e17fe5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:03.018694 17546 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc36fd6556344159f0d8419e17fe5c1 (DEFAULT_TABLE table=test-table [id=5e8c29a5297b459e853b3583de08a98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:03.019016 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc36fd6556344159f0d8419e17fe5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:03.019016 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.012800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.0.98.2:44175)
0620 14:30:03.012981 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:03.012987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:03.012996 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:30:03.013036 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:03.013038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:03.013055 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:03.013145 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:03.013358 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.014574 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.016876 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.017644 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.017713 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.017805 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:03.018234 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 14:30:03.018920 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
I20260620 14:30:03.019672 17677 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc36fd6556344159f0d8419e17fe5c1 (DEFAULT_TABLE table=test-table [id=5e8c29a5297b459e853b3583de08a98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:03.019986 17677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc36fd6556344159f0d8419e17fe5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:03.021771 17888 tablet_bootstrap.cc:492] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6: Bootstrap starting.
I20260620 14:30:03.022485 17888 tablet_bootstrap.cc:654] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:03.022517 17889 tablet_bootstrap.cc:492] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42: Bootstrap starting.
I20260620 14:30:03.022810 17888 log.cc:824] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:03.023175 17889 tablet_bootstrap.cc:654] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:03.023288 17891 tablet_bootstrap.cc:492] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161: Bootstrap starting.
I20260620 14:30:03.023510 17889 log.cc:824] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:03.023659 17888 tablet_bootstrap.cc:492] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6: No bootstrap required, opened a new log
I20260620 14:30:03.023716 17888 ts_tablet_manager.cc:1397] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:30:03.023914 17891 tablet_bootstrap.cc:654] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:03.024158 17889 tablet_bootstrap.cc:492] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42: No bootstrap required, opened a new log
I20260620 14:30:03.024199 17889 ts_tablet_manager.cc:1397] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:03.024197 17891 log.cc:824] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:03.025125 17891 tablet_bootstrap.cc:492] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161: No bootstrap required, opened a new log
I20260620 14:30:03.025197 17891 ts_tablet_manager.cc:1397] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:03.025941 17888 raft_consensus.cc:348] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b554428e5c4fa995c468f8476321e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46471 } } peers { permanent_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44175 } } peers { permanent_uuid: "00793372fd89498586b42b3fa357db42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260620 14:30:03.026094 17888 raft_consensus.cc:374] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:03.026137 17888 raft_consensus.cc:729] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b554428e5c4fa995c468f8476321e6, State: Initialized, Role: FOLLOWER
I20260620 14:30:03.026248 17888 consensus_queue.cc:260] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b554428e5c4fa995c468f8476321e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46471 } } peers { permanent_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44175 } } peers { permanent_uuid: "00793372fd89498586b42b3fa357db42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260620 14:30:03.026578 17874 heartbeater.cc:503] Master 127.0.98.62:46055 was elected leader, sending a full tablet report...
I20260620 14:30:03.026569 17888 ts_tablet_manager.cc:1428] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:30:03.026813 17889 raft_consensus.cc:348] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b554428e5c4fa995c468f8476321e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46471 } } peers { permanent_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44175 } } peers { permanent_uuid: "00793372fd89498586b42b3fa357db42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260620 14:30:03.026975 17889 raft_consensus.cc:374] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:03.027017 17889 raft_consensus.cc:729] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00793372fd89498586b42b3fa357db42, State: Initialized, Role: FOLLOWER
I20260620 14:30:03.027130 17889 consensus_queue.cc:260] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b554428e5c4fa995c468f8476321e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46471 } } peers { permanent_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44175 } } peers { permanent_uuid: "00793372fd89498586b42b3fa357db42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260620 14:30:03.027379 17889 ts_tablet_manager.cc:1428] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:03.027402 17894 raft_consensus.cc:482] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:03.027495 17612 heartbeater.cc:503] Master 127.0.98.62:46055 was elected leader, sending a full tablet report...
I20260620 14:30:03.027477 17894 raft_consensus.cc:504] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b554428e5c4fa995c468f8476321e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46471 } } peers { permanent_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44175 } } peers { permanent_uuid: "00793372fd89498586b42b3fa357db42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260620 14:30:03.027757 17894 leader_election.cc:288] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e7ac5a45f84423a9b46f60ca5d2e161 (127.0.98.2:44175), 00793372fd89498586b42b3fa357db42 (127.0.98.1:43817)
I20260620 14:30:03.027729 17891 raft_consensus.cc:348] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b554428e5c4fa995c468f8476321e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46471 } } peers { permanent_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44175 } } peers { permanent_uuid: "00793372fd89498586b42b3fa357db42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260620 14:30:03.027877 17891 raft_consensus.cc:374] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:03.027915 17891 raft_consensus.cc:729] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7ac5a45f84423a9b46f60ca5d2e161, State: Initialized, Role: FOLLOWER
I20260620 14:30:03.028026 17891 consensus_queue.cc:260] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b554428e5c4fa995c468f8476321e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46471 } } peers { permanent_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44175 } } peers { permanent_uuid: "00793372fd89498586b42b3fa357db42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260620 14:30:03.028283 17891 ts_tablet_manager.cc:1428] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:03.028401 17743 heartbeater.cc:503] Master 127.0.98.62:46055 was elected leader, sending a full tablet report...
I20260620 14:30:03.031515 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.027963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44175 (local address 127.0.98.3:44453)
0620 14:30:03.028061 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:03.028071 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:03.028120 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:03.028308 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:03.028309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:03.028311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:03.028377 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:03.028379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.029304 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.029304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:03.030508 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:03.030510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.031253 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.031254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:03.031308 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.031312 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:30:03.031389 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:30:03.031432 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 14:30:03.031529 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.028027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44453 (local address 127.0.98.2:44175)
0620 14:30:03.028065 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:03.028067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:03.028106 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:30:03.028175 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:03.028176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:03.028198 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:03.028255 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:03.028446 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.029183 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.030582 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.031176 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.031248 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.031258 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:03.031405 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 14:30:03.031441 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:30:03.031607 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.028496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56389 (local address 127.0.98.1:43817)
0620 14:30:03.028535 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:03.028537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:03.028545 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:30:03.028565 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:03.028565 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:03.028574 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:03.028621 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:03.028793 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.029443 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.030740 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.031279 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.031339 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.031349 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:03.031503 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:30:03.031537 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:30:03.031635 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.027970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43817 (local address 127.0.98.3:56389)
0620 14:30:03.028285 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:03.028308 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:03.028352 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:03.028660 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:03.028661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:03.028663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:03.028721 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:03.028723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.029525 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.029526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:03.030628 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:03.030630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.031360 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.031361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:03.031434 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.031437 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:03.031488 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:30:03.031514 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":123,"thread_start_us":77,"threads_started":1}
I20260620 14:30:03.031958 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc36fd6556344159f0d8419e17fe5c1" candidate_uuid: "13b554428e5c4fa995c468f8476321e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00793372fd89498586b42b3fa357db42" is_pre_election: true
I20260620 14:30:03.032053 17566 raft_consensus.cc:2454] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b554428e5c4fa995c468f8476321e6 in term 0.
I20260620 14:30:03.032225 17697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc36fd6556344159f0d8419e17fe5c1" candidate_uuid: "13b554428e5c4fa995c468f8476321e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161" is_pre_election: true
I20260620 14:30:03.032265 17762 leader_election.cc:302] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00793372fd89498586b42b3fa357db42, 13b554428e5c4fa995c468f8476321e6; no voters:
I20260620 14:30:03.032357 17697 raft_consensus.cc:2454] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b554428e5c4fa995c468f8476321e6 in term 0.
I20260620 14:30:03.032384 17894 raft_consensus.cc:2781] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:03.032437 17894 raft_consensus.cc:482] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:03.032461 17894 raft_consensus.cc:3037] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:03.033087 17894 raft_consensus.cc:504] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b554428e5c4fa995c468f8476321e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46471 } } peers { permanent_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44175 } } peers { permanent_uuid: "00793372fd89498586b42b3fa357db42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260620 14:30:03.033236 17894 leader_election.cc:288] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [CANDIDATE]: Term 1 election: Requested vote from peers 9e7ac5a45f84423a9b46f60ca5d2e161 (127.0.98.2:44175), 00793372fd89498586b42b3fa357db42 (127.0.98.1:43817)
I20260620 14:30:03.033382 17697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc36fd6556344159f0d8419e17fe5c1" candidate_uuid: "13b554428e5c4fa995c468f8476321e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161"
I20260620 14:30:03.033414 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc36fd6556344159f0d8419e17fe5c1" candidate_uuid: "13b554428e5c4fa995c468f8476321e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00793372fd89498586b42b3fa357db42"
I20260620 14:30:03.033454 17697 raft_consensus.cc:3037] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:03.033515 17566 raft_consensus.cc:3037] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:03.034160 17697 raft_consensus.cc:2454] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b554428e5c4fa995c468f8476321e6 in term 1.
I20260620 14:30:03.034227 17566 raft_consensus.cc:2454] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b554428e5c4fa995c468f8476321e6 in term 1.
I20260620 14:30:03.034374 17763 leader_election.cc:302] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b554428e5c4fa995c468f8476321e6, 9e7ac5a45f84423a9b46f60ca5d2e161; no voters:
I20260620 14:30:03.034474 17894 raft_consensus.cc:2781] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:03.034631 17894 raft_consensus.cc:686] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 1 LEADER]: Becoming Leader. State: Replica: 13b554428e5c4fa995c468f8476321e6, State: Running, Role: LEADER
I20260620 14:30:03.034736 17894 consensus_queue.cc:237] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b554428e5c4fa995c468f8476321e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46471 } } peers { permanent_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44175 } } peers { permanent_uuid: "00793372fd89498586b42b3fa357db42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260620 14:30:03.035487 17424 catalog_manager.cc:5495] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13b554428e5c4fa995c468f8476321e6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "13b554428e5c4fa995c468f8476321e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b554428e5c4fa995c468f8476321e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00793372fd89498586b42b3fa357db42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } health_report { overall_health: UNKNOWN } } }
I20260620 14:30:03.044678 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.042281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.0.98.3:46471)
0620 14:30:03.042355 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:03.042358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:03.042367 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:30:03.042448 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:03.042449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:03.042469 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:03.042545 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:03.042648 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.043104 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.043894 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.044083 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.044161 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.044344 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:03.044437 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:03.044534 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:03.044554 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 14:30:03.044572 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:30:03.046475 17566 raft_consensus.cc:1261] T 5cc36fd6556344159f0d8419e17fe5c1 P 00793372fd89498586b42b3fa357db42 [term 1 FOLLOWER]: Refusing update from remote peer 13b554428e5c4fa995c468f8476321e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:03.046564 17697 raft_consensus.cc:1261] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161 [term 1 FOLLOWER]: Refusing update from remote peer 13b554428e5c4fa995c468f8476321e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:03.046850 17894 consensus_queue.cc:1035] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00793372fd89498586b42b3fa357db42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:03.046979 17898 consensus_queue.cc:1035] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e7ac5a45f84423a9b46f60ca5d2e161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:03.049589 17901 mvcc.cc:205] Tried to move back new op lower bound from 7298931929275543552 to 7298931929230991360. Current Snapshot: MvccSnapshot[applied={T|T < 7298931929275543552}]
I20260620 14:30:03.049752 17903 mvcc.cc:205] Tried to move back new op lower bound from 7298931929275543552 to 7298931929230991360. Current Snapshot: MvccSnapshot[applied={T|T < 7298931929275543552}]
I20260620 14:30:03.050176 17902 mvcc.cc:205] Tried to move back new op lower bound from 7298931929275543552 to 7298931929230991360. Current Snapshot: MvccSnapshot[applied={T|T < 7298931929275543552}]
I20260620 14:30:03.052431 17423 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:03.052568 17423 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:03.053393 17423 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384c20c9ea7e45f1946849e352a29073: Sending DeleteTablet for 3 replicas of tablet 5cc36fd6556344159f0d8419e17fe5c1
I20260620 14:30:03.053754 17808 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc36fd6556344159f0d8419e17fe5c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:03 UTC) from {username='slave'} at 127.0.0.1:57324
I20260620 14:30:03.053836 17677 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc36fd6556344159f0d8419e17fe5c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:03 UTC) from {username='slave'} at 127.0.0.1:33666
I20260620 14:30:03.053864 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc36fd6556344159f0d8419e17fe5c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:03 UTC) from {username='slave'} at 127.0.0.1:58292
I20260620 14:30:03.054323 17910 tablet_replica.cc:330] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6: stopping tablet replica
I20260620 14:30:03.054455 17911 tablet_replica.cc:330] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161: stopping tablet replica
I20260620 14:30:03.054491 17910 raft_consensus.cc:2229] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:30:03.054504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 17484
I20260620 14:30:03.056037 17910 raft_consensus.cc:2258] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:03.056521 17910 ts_tablet_manager.cc:1905] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:03.058755 17910 ts_tablet_manager.cc:1918] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:03.058838 17910 log.cc:1196] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/5cc36fd6556344159f0d8419e17fe5c1
I20260620 14:30:03.054576 17911 raft_consensus.cc:2229] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:03.059108 17910 ts_tablet_manager.cc:1939] T 5cc36fd6556344159f0d8419e17fe5c1 P 13b554428e5c4fa995c468f8476321e6: Deleting consensus metadata
I20260620 14:30:03.059291 17911 raft_consensus.cc:2258] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:03.059538 17410 catalog_manager.cc:4841] TS 13b554428e5c4fa995c468f8476321e6 (127.0.98.3:46471): tablet 5cc36fd6556344159f0d8419e17fe5c1 (table test-table [id=5e8c29a5297b459e853b3583de08a98e]) successfully deleted
I20260620 14:30:03.059998 17911 ts_tablet_manager.cc:1905] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:30:03.061246 17409 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43817 (error 108)
I20260620 14:30:03.061518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 17615
I20260620 14:30:03.061650 17911 ts_tablet_manager.cc:1918] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:03.061731 17911 log.cc:1196] T 5cc36fd6556344159f0d8419e17fe5c1 P 9e7ac5a45f84423a9b46f60ca5d2e161: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/5cc36fd6556344159f0d8419e17fe5c1
I20260620 14:30:03.062144 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.061764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43817 (local address 127.0.0.1:58308)
0620 14:30:03.062054 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:03.062099 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263}
W20260620 14:30:03.062415 17409 catalog_manager.cc:4584] TS 00793372fd89498586b42b3fa357db42 (127.0.98.1:43817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cc36fd6556344159f0d8419e17fe5c1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43817: connect: Connection refused (error 111)
I20260620 14:30:03.067812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 17746
I20260620 14:30:03.067893 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.067762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44175 (local address 127.0.0.1:33668)
0620 14:30:03.067822 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:03.067849 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:30:03.072572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 17393
2026-06-20T14:30:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (681 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:03 dist-test-slave-hfc3 krb5kdc[17916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:03 dist-test-slave-hfc3 krb5kdc[17916](info): set up 2 sockets
Jun 20 14:30:03 dist-test-slave-hfc3 krb5kdc[17916](info): commencing operation
krb5kdc: starting...
W20260620 14:30:04.170127 392 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:04 dist-test-slave-hfc3 krb5kdc[17916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:04.295006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34309
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:04.375571 17929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:04.375780 17929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:04.375802 17929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:04.376881 17929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:04.376916 17929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:04.376930 17929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:04.376943 17929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:04.376955 17929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:04.378364 17929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:04.378547 17929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:04.378721 17929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:04.383352 17937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:04.383391 17935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:04.383504 17934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:04.383697 17929 server_base.cc:1034] running on GCE node
I20260620 14:30:04.383961 17929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:04.384249 17929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:04.385414 17929 hybrid_clock.cc:648] HybridClock initialized: now 1781965804385394 us; error 31 us; skew 500 ppm
Jun 20 14:30:04 dist-test-slave-hfc3 krb5kdc[17916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:04.387910 17929 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:04.388376 17929 webserver.cc:460] Webserver started at http://127.0.98.62:45953/ using document root <none> and password file <none>
I20260620 14:30:04.388618 17929 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:04.388687 17929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:04.388842 17929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:04.389880 17929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "020c63be79e847d2815464f1d7f30d3d"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-hfc3"
server_key: "fbfe8e47f94f66b1f0db7d338baf6fa6"
server_key_iv: "ac77a0c340f5d0eedfcb6b4645f2369a"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.390256 17929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "020c63be79e847d2815464f1d7f30d3d"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-hfc3"
server_key: "fbfe8e47f94f66b1f0db7d338baf6fa6"
server_key_iv: "ac77a0c340f5d0eedfcb6b4645f2369a"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.391688 17929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:04.392396 17944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:04.392567 17929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:04.392671 17929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "020c63be79e847d2815464f1d7f30d3d"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-hfc3"
server_key: "fbfe8e47f94f66b1f0db7d338baf6fa6"
server_key_iv: "ac77a0c340f5d0eedfcb6b4645f2369a"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.392745 17929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:04.413663 17929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:04.415757 17929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:04.415938 17929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:04.420224 17929 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34309
I20260620 14:30:04.420300 17996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34309 every 8 connection(s)
I20260620 14:30:04.420666 17929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:04.421514 17997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:04.423942 17997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d: Bootstrap starting.
I20260620 14:30:04.424563 17997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:04.424868 17997 log.cc:824] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:04.425568 17997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d: No bootstrap required, opened a new log
I20260620 14:30:04.427289 17997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020c63be79e847d2815464f1d7f30d3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } }
I20260620 14:30:04.427434 17997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:04.427457 17997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 020c63be79e847d2815464f1d7f30d3d, State: Initialized, Role: FOLLOWER
I20260620 14:30:04.427632 17997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020c63be79e847d2815464f1d7f30d3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } }
I20260620 14:30:04.427699 17997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:04.427721 17997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:04.427793 17997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:04.428365 17997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020c63be79e847d2815464f1d7f30d3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } }
I20260620 14:30:04.428480 17997 leader_election.cc:302] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 020c63be79e847d2815464f1d7f30d3d; no voters:
I20260620 14:30:04.428670 17997 leader_election.cc:288] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:04.428783 18000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:04.428961 17997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:04.428978 18000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [term 1 LEADER]: Becoming Leader. State: Replica: 020c63be79e847d2815464f1d7f30d3d, State: Running, Role: LEADER
I20260620 14:30:04.429113 18000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020c63be79e847d2815464f1d7f30d3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } }
I20260620 14:30:04.429636 18002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 020c63be79e847d2815464f1d7f30d3d. Latest consensus state: current_term: 1 leader_uuid: "020c63be79e847d2815464f1d7f30d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020c63be79e847d2815464f1d7f30d3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } } }
I20260620 14:30:04.429733 18002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:04.429731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 17929
I20260620 14:30:04.429845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:04.429910 18001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "020c63be79e847d2815464f1d7f30d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020c63be79e847d2815464f1d7f30d3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } } }
I20260620 14:30:04.429986 18001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:04.430047 392 external_mini_cluster.cc:1428] Setting key d1d4a46dd3654c9bdaf15719a185458c
I20260620 14:30:04.430105 18009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:04.430630 18009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:04.432180 18009 catalog_manager.cc:1347] Generated new cluster ID: ad63386595414254bf9ac7a38fd82afd
I20260620 14:30:04.432232 18009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:30:04 dist-test-slave-hfc3 krb5kdc[17916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:04.441074 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.430746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.0.98.62:34309)
0620 14:30:04.431310 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:04.431317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:04.431347 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:30:04.431439 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:04.431444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:04.431487 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:04.431653 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:04.432484 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.433411 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.434203 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.434730 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.434855 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.437274 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:04.437279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:04.437280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:04.437292 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:04.439404 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:04.439794 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:04.439797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:04.439798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:04.439912 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:04.440135 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:04.440136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:04.440137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:04.440402 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:04.440581 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:04.440695 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:30:04.440865 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":512,"thread_start_us":75,"threads_started":1}
I20260620 14:30:04.455794 18009 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:04.456347 18009 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:04.462667 18009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d: Generated new TSK 0
I20260620 14:30:04.462888 18009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:30:04.498440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34309
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:04.583475 18023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:04.583686 18023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:04.583710 18023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:04.584900 18023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:04.584944 18023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:04.584995 18023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:30:04.586555 18023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:04.586794 18023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:04.587033 18023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:04.592485 18031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:04.592549 18023 server_base.cc:1034] running on GCE node
W20260620 14:30:04.592481 18029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:04.592476 18028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:04.592931 18023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:04.593230 18023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:04.594412 18023 hybrid_clock.cc:648] HybridClock initialized: now 1781965804594387 us; error 34 us; skew 500 ppm
Jun 20 14:30:04 dist-test-slave-hfc3 krb5kdc[17916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:04.597368 18023 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:30:04.597908 18023 webserver.cc:460] Webserver started at http://127.0.98.1:45981/ using document root <none> and password file <none>
I20260620 14:30:04.598153 18023 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:04.598229 18023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:04.598423 18023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:04.599628 18023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "25b52231eaf040adbfae1375946a00f5"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-hfc3"
server_key: "4c1ff91ffefedf62ca2d5c5a04b4dc40"
server_key_iv: "fce1322609227e690d0eb8f50ed83442"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.600029 18023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "25b52231eaf040adbfae1375946a00f5"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-hfc3"
server_key: "4c1ff91ffefedf62ca2d5c5a04b4dc40"
server_key_iv: "fce1322609227e690d0eb8f50ed83442"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.601495 18023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:04.602267 18038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:04.602488 18023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:04.602602 18023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "25b52231eaf040adbfae1375946a00f5"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-hfc3"
server_key: "4c1ff91ffefedf62ca2d5c5a04b4dc40"
server_key_iv: "fce1322609227e690d0eb8f50ed83442"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.602676 18023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:04.613513 18023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:04.615926 18023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:04.616092 18023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:04.616328 18023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:04.616649 18023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:04.616701 18023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:04.616744 18023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:04.616780 18023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:04.622856 18023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46707
I20260620 14:30:04.622965 18151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46707 every 8 connection(s)
I20260620 14:30:04.623260 18023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:04.623636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 18023
I20260620 14:30:04.623744 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:04.623967 392 external_mini_cluster.cc:1428] Setting key 6635d335d4d4f548e00776702e9ef66a
Jun 20 14:30:04 dist-test-slave-hfc3 krb5kdc[17916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:30:04.634014 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.624164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49033 (local address 127.0.98.62:34309)
0620 14:30:04.624215 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:04.624218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:04.624361 (+ 143us) server_negotiation.cc:407] Connection header received
0620 14:30:04.625678 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:04.625679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:04.625688 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:04.625726 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:04.626691 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.627204 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.628250 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.628537 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.628634 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.631066 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:04.631068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:04.631069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:04.631076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:04.632525 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:04.632945 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:04.632947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:04.632947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:04.633014 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:04.633279 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:04.633281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:04.633282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:04.633431 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:04.633509 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:04.633802 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 14:30:04.633921 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:30:04.634168 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.624001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34309 (local address 127.0.98.1:49033)
0620 14:30:04.624218 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:04.624263 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:04.625556 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:04.625809 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:04.625814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:04.626247 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:04.626584 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.626591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.627305 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.627307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.628152 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.628154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.628638 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.628639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.628784 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.629430 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:04.629438 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:04.630962 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:04.632625 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:04.632632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:04.632635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:04.632865 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:04.633083 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:04.633084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:04.633085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:04.633203 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:04.633500 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:04.633506 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:04.633655 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 14:30:04.633761 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:30:04.633935 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":105,"threads_started":1}
I20260620 14:30:04.634681 18152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34309
I20260620 14:30:04.634796 18152 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:30:04.635010 18152 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
I20260620 14:30:04.635519 17961 ts_manager.cc:194] Registered new tserver with Master: 25b52231eaf040adbfae1375946a00f5 (127.0.98.1:46707)
I20260620 14:30:04.636142 17961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:30:04.649823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34309
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:04.732172 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:04.732396 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:04.732421 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:04.733528 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:04.733566 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:04.733592 18157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:30:04.735149 18157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:04.735340 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:04.735515 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:04.740962 18165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:04.740957 18163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:04.740957 18162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:04.741113 18157 server_base.cc:1034] running on GCE node
I20260620 14:30:04.741421 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:04.741693 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:04.742849 18157 hybrid_clock.cc:648] HybridClock initialized: now 1781965804742847 us; error 38 us; skew 500 ppm
Jun 20 14:30:04 dist-test-slave-hfc3 krb5kdc[17916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:04.745383 18157 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:30:04.745774 18157 webserver.cc:460] Webserver started at http://127.0.98.2:44629/ using document root <none> and password file <none>
I20260620 14:30:04.745965 18157 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:04.746007 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:04.746098 18157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:04.747217 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "8e23c6372ee844a7b2c3f543ccf016e0"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-hfc3"
server_key: "0ee227be33525ecf69df4deb712687b5"
server_key_iv: "0b5b078a510f25228301e38867bc8e48"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.747531 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8e23c6372ee844a7b2c3f543ccf016e0"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-hfc3"
server_key: "0ee227be33525ecf69df4deb712687b5"
server_key_iv: "0b5b078a510f25228301e38867bc8e48"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.748858 18157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:30:04.749581 18172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:04.749837 18157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:30:04.749938 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "8e23c6372ee844a7b2c3f543ccf016e0"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-hfc3"
server_key: "0ee227be33525ecf69df4deb712687b5"
server_key_iv: "0b5b078a510f25228301e38867bc8e48"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.749998 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:04.772423 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:04.774901 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:04.775074 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:04.775357 18157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:04.775625 18157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:04.775657 18157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:04.775686 18157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:04.775698 18157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:04.781474 18157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37721
I20260620 14:30:04.781518 18285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37721 every 8 connection(s)
I20260620 14:30:04.781850 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:04.785327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 18157
I20260620 14:30:04.785421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:04.785632 392 external_mini_cluster.cc:1428] Setting key 24c80d94197874e543f567c15b0cad9f
Jun 20 14:30:04 dist-test-slave-hfc3 krb5kdc[17916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:04.792547 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.782592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34309 (local address 127.0.98.2:37439)
0620 14:30:04.782792 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:04.782829 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:04.783787 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:04.784073 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:04.784076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:04.784345 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:04.784536 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.784540 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.785245 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.785247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.786505 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.786508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.786910 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.786911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.787044 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.787606 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:04.787614 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:04.789307 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:04.791159 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:04.791165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:04.791167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:04.791391 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:04.791645 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:04.791645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:04.791646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:04.791746 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:04.792066 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:04.792070 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:04.792185 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 14:30:04.792268 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:30:04.792385 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260620 14:30:04.792547 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.782646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37439 (local address 127.0.98.62:34309)
0620 14:30:04.782711 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:04.782713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:04.782925 (+ 212us) server_negotiation.cc:407] Connection header received
0620 14:30:04.783907 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:04.783909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:04.783922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:04.783984 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:04.784636 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.785137 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.786588 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.786788 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.786902 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.789407 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:04.789409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:04.789409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:04.789417 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:04.791062 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:04.791500 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:04.791501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:04.791501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:04.791573 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:04.791831 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:04.791832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:04.791832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:04.791993 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:04.792075 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:04.792316 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 14:30:04.792408 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:30:04.793205 18286 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34309
I20260620 14:30:04.793341 18286 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:04.793520 18286 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
I20260620 14:30:04.793932 17961 ts_manager.cc:194] Registered new tserver with Master: 8e23c6372ee844a7b2c3f543ccf016e0 (127.0.98.2:37721)
I20260620 14:30:04.794312 17961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37439
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:30:04.811244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34309
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:04.891798 18291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:04.892053 18291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:04.892093 18291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:04.893265 18291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:04.893339 18291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:04.893380 18291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:30:04.894954 18291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:04.895221 18291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:04.895453 18291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:30:04.900638 18291 server_base.cc:1034] running on GCE node
W20260620 14:30:04.900755 18297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:04.900651 18296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:04.900810 18299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:04.901063 18291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:04.901337 18291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:04.902510 18291 hybrid_clock.cc:648] HybridClock initialized: now 1781965804902486 us; error 40 us; skew 500 ppm
Jun 20 14:30:04 dist-test-slave-hfc3 krb5kdc[17916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:04.905076 18291 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:30:04.905529 18291 webserver.cc:460] Webserver started at http://127.0.98.3:45191/ using document root <none> and password file <none>
I20260620 14:30:04.905838 18291 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:04.905887 18291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:04.906018 18291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:04.907229 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "31957a5ddd3a4cd3ab282f05726190d9"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-hfc3"
server_key: "2b6b6b3cec969a8cc3bd204dc345b9bd"
server_key_iv: "db181c38193ddaf3fed7b75570147640"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.907596 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "31957a5ddd3a4cd3ab282f05726190d9"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-hfc3"
server_key: "2b6b6b3cec969a8cc3bd204dc345b9bd"
server_key_iv: "db181c38193ddaf3fed7b75570147640"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.908921 18291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:30:04.909713 18306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:04.909902 18291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:30:04.910002 18291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "31957a5ddd3a4cd3ab282f05726190d9"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-hfc3"
server_key: "2b6b6b3cec969a8cc3bd204dc345b9bd"
server_key_iv: "db181c38193ddaf3fed7b75570147640"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.910089 18291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:04.930533 18291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:04.932750 18291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:04.932947 18291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:04.933192 18291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:04.933534 18291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:04.933578 18291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:04.933601 18291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:04.933650 18291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:04.940061 18291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34523
I20260620 14:30:04.940138 18419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34523 every 8 connection(s)
I20260620 14:30:04.940479 18291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:04.946492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 18291
I20260620 14:30:04.946610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:04.946815 392 external_mini_cluster.cc:1428] Setting key 01414116c6bcb0a6e9970a67e96f9397
Jun 20 14:30:04 dist-test-slave-hfc3 krb5kdc[17916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:04.950553 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.941235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51695 (local address 127.0.98.62:34309)
0620 14:30:04.941326 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:04.941328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:04.941532 (+ 204us) server_negotiation.cc:407] Connection header received
0620 14:30:04.942517 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:04.942518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:04.942534 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:04.942595 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:04.943203 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.943710 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.944795 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.944987 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.945052 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.947732 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:04.947734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:04.947734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:04.947741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:04.948997 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:04.949465 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:04.949466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:04.949466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:04.949529 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:04.949809 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:04.949810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:04.949811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:04.949935 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:04.950023 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:04.950372 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 14:30:04.950461 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:30:04.950766 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.941195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34309 (local address 127.0.98.3:51695)
0620 14:30:04.941400 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:04.941448 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:04.942403 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:04.942657 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:04.942660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:04.942928 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:04.943118 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.943123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.943790 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.943791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.944615 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.944616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.945069 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.945070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.945168 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.945787 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:04.945796 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:04.947639 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:04.949113 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:04.949121 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:04.949123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:04.949371 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:04.949597 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:04.949598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:04.949599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:04.949736 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:04.950014 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:04.950020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:04.950202 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 14:30:04.950312 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:30:04.950502 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":120,"threads_started":1}
I20260620 14:30:04.951225 18420 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34309
I20260620 14:30:04.951323 18420 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:04.951506 18420 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
I20260620 14:30:04.951996 17961 ts_manager.cc:194] Registered new tserver with Master: 31957a5ddd3a4cd3ab282f05726190d9 (127.0.98.3:34523)
I20260620 14:30:04.952433 17961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51695
I20260620 14:30:04.958736 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:30:04.966989 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.960070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.0.98.62:34309)
0620 14:30:04.960159 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:04.960162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:04.960281 (+ 119us) server_negotiation.cc:407] Connection header received
0620 14:30:04.960364 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:04.960365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:04.960375 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:04.960448 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:04.961060 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.961554 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.962303 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.962556 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.962637 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.964200 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:04.964201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:04.964202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:04.964210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:04.965665 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:04.966088 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:04.966089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:04.966089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:04.966155 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:04.966407 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:04.966408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:04.966408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:04.966558 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:04.966674 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:04.966822 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:30:04.966885 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:30:04.968858 17961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:04.970528 17961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:04.977095 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.973322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.0.98.1:46707)
0620 14:30:04.973511 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:04.973517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:04.973526 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:30:04.973564 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:04.973566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:04.973581 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:04.973657 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:04.973763 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.974894 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.976137 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.976653 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.976750 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.976823 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:04.976972 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 14:30:04.977012 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260620 14:30:04.977129 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.973075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46707 (local address 127.0.0.1:54522)
0620 14:30:04.973222 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:04.973230 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:04.973265 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:04.973666 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:04.973667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:04.973669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:04.973705 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.973706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.974994 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.974995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.976051 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.976053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.976752 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.976753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.976841 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.976855 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:30:04.976954 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:30:04.977009 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260620 14:30:04.977303 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.972987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.0.98.2:37721)
0620 14:30:04.973340 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:04.973345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:04.973358 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:30:04.973401 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:04.973403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:04.973420 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:04.973508 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:04.973710 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.974588 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.976040 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.976794 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.976868 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.976927 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:04.977172 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:30:04.977217 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":83,"threads_started":1}
I20260620 14:30:04.977344 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.972827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34523 (local address 127.0.0.1:45182)
0620 14:30:04.973191 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:04.973203 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:04.973275 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:04.973596 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:04.973597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:04.973600 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:04.973845 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.973847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.975108 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.975109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.975992 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.975995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.976898 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.976899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.976965 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.976968 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:04.976999 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 14:30:04.977042 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":6,"thread_start_us":98,"threads_started":1}
I20260620 14:30:04.977367 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.973047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.0.98.3:34523)
0620 14:30:04.973400 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:04.973404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:04.973431 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:30:04.973472 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:04.973474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:04.973490 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:04.973922 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:04.974004 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.975012 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.976088 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.976802 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.977157 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.977217 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:04.977240 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 14:30:04.977278 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":103,"threads_started":1}
I20260620 14:30:04.977492 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.972827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37721 (local address 127.0.0.1:41578)
0620 14:30:04.972981 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:04.973024 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:04.973098 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:04.973512 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:04.973516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:04.973518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:04.973600 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.973602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.974686 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.974687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.975950 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.975952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.976956 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.976957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.977015 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.977017 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:30:04.977061 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:30:04.977097 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":102,"threads_started":1}
I20260620 14:30:04.978065 18354 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6763bb626c443d866821d5e67bc8f1 (DEFAULT_TABLE table=test-table [id=7ed4575048f84823b68b0d958de39e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:04.978263 18220 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6763bb626c443d866821d5e67bc8f1 (DEFAULT_TABLE table=test-table [id=7ed4575048f84823b68b0d958de39e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:04.978367 18086 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6763bb626c443d866821d5e67bc8f1 (DEFAULT_TABLE table=test-table [id=7ed4575048f84823b68b0d958de39e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:04.978479 18354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6763bb626c443d866821d5e67bc8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:04.978639 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6763bb626c443d866821d5e67bc8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:04.978682 18086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6763bb626c443d866821d5e67bc8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:04.981204 18437 tablet_bootstrap.cc:492] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0: Bootstrap starting.
I20260620 14:30:04.981801 18438 tablet_bootstrap.cc:492] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5: Bootstrap starting.
I20260620 14:30:04.981973 18437 tablet_bootstrap.cc:654] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:04.982506 18438 tablet_bootstrap.cc:654] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:04.982496 18439 tablet_bootstrap.cc:492] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9: Bootstrap starting.
I20260620 14:30:04.982657 18437 log.cc:824] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:04.983052 18438 log.cc:824] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:04.983129 18439 tablet_bootstrap.cc:654] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:04.983522 18437 tablet_bootstrap.cc:492] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0: No bootstrap required, opened a new log
I20260620 14:30:04.983601 18437 ts_tablet_manager.cc:1397] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:04.983705 18439 log.cc:824] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:04.983798 18438 tablet_bootstrap.cc:492] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5: No bootstrap required, opened a new log
I20260620 14:30:04.983868 18438 ts_tablet_manager.cc:1397] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:30:04.984477 18439 tablet_bootstrap.cc:492] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9: No bootstrap required, opened a new log
I20260620 14:30:04.984546 18439 ts_tablet_manager.cc:1397] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:04.985469 18438 raft_consensus.cc:348] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e23c6372ee844a7b2c3f543ccf016e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37721 } } peers { permanent_uuid: "31957a5ddd3a4cd3ab282f05726190d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } } peers { permanent_uuid: "25b52231eaf040adbfae1375946a00f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46707 } }
I20260620 14:30:04.985634 18438 raft_consensus.cc:374] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:04.985678 18438 raft_consensus.cc:729] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b52231eaf040adbfae1375946a00f5, State: Initialized, Role: FOLLOWER
I20260620 14:30:04.985822 18438 consensus_queue.cc:260] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e23c6372ee844a7b2c3f543ccf016e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37721 } } peers { permanent_uuid: "31957a5ddd3a4cd3ab282f05726190d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } } peers { permanent_uuid: "25b52231eaf040adbfae1375946a00f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46707 } }
I20260620 14:30:04.985709 18437 raft_consensus.cc:348] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e23c6372ee844a7b2c3f543ccf016e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37721 } } peers { permanent_uuid: "31957a5ddd3a4cd3ab282f05726190d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } } peers { permanent_uuid: "25b52231eaf040adbfae1375946a00f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46707 } }
I20260620 14:30:04.985909 18437 raft_consensus.cc:374] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:04.985954 18437 raft_consensus.cc:729] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e23c6372ee844a7b2c3f543ccf016e0, State: Initialized, Role: FOLLOWER
I20260620 14:30:04.986096 18438 ts_tablet_manager.cc:1428] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:30:04.986078 18437 consensus_queue.cc:260] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e23c6372ee844a7b2c3f543ccf016e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37721 } } peers { permanent_uuid: "31957a5ddd3a4cd3ab282f05726190d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } } peers { permanent_uuid: "25b52231eaf040adbfae1375946a00f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46707 } }
I20260620 14:30:04.986236 18152 heartbeater.cc:503] Master 127.0.98.62:34309 was elected leader, sending a full tablet report...
I20260620 14:30:04.986370 18437 ts_tablet_manager.cc:1428] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:04.986493 18286 heartbeater.cc:503] Master 127.0.98.62:34309 was elected leader, sending a full tablet report...
I20260620 14:30:04.986922 18439 raft_consensus.cc:348] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e23c6372ee844a7b2c3f543ccf016e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37721 } } peers { permanent_uuid: "31957a5ddd3a4cd3ab282f05726190d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } } peers { permanent_uuid: "25b52231eaf040adbfae1375946a00f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46707 } }
I20260620 14:30:04.987109 18439 raft_consensus.cc:374] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:04.987155 18439 raft_consensus.cc:729] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31957a5ddd3a4cd3ab282f05726190d9, State: Initialized, Role: FOLLOWER
I20260620 14:30:04.987279 18439 consensus_queue.cc:260] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e23c6372ee844a7b2c3f543ccf016e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37721 } } peers { permanent_uuid: "31957a5ddd3a4cd3ab282f05726190d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } } peers { permanent_uuid: "25b52231eaf040adbfae1375946a00f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46707 } }
I20260620 14:30:04.987558 18439 ts_tablet_manager.cc:1428] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:04.987584 18420 heartbeater.cc:503] Master 127.0.98.62:34309 was elected leader, sending a full tablet report...
I20260620 14:30:04.994071 18443 raft_consensus.cc:482] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:04.994158 18443 raft_consensus.cc:504] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e23c6372ee844a7b2c3f543ccf016e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37721 } } peers { permanent_uuid: "31957a5ddd3a4cd3ab282f05726190d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } } peers { permanent_uuid: "25b52231eaf040adbfae1375946a00f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46707 } }
I20260620 14:30:04.994488 18443 leader_election.cc:288] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e23c6372ee844a7b2c3f543ccf016e0 (127.0.98.2:37721), 31957a5ddd3a4cd3ab282f05726190d9 (127.0.98.3:34523)
I20260620 14:30:04.997541 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.994708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37721 (local address 127.0.98.1:49803)
0620 14:30:04.994845 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:04.994857 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:04.994909 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:04.995038 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:04.995039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:04.995041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:04.995112 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.995114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.995855 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.995856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.996685 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.996687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.997275 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.997276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.997369 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.997374 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:30:04.997429 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:30:04.997470 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260620 14:30:04.997601 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.994797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49803 (local address 127.0.98.2:37721)
0620 14:30:04.994907 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:04.994909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:04.994920 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:30:04.994960 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:04.994961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:04.994973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:04.995033 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:04.995187 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.995776 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.996759 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.997196 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.997255 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.997269 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:04.997466 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 14:30:04.997513 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 14:30:04.997771 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.994796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52067 (local address 127.0.98.3:34523)
0620 14:30:04.994862 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:04.994864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:04.994971 (+ 107us) server_negotiation.cc:407] Connection header received
0620 14:30:04.995023 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:04.995024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:04.995033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:04.995087 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:04.995285 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.995784 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.996931 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.997363 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.997437 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.997446 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:04.997664 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:30:04.997706 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:30:04.997999 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.994713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34523 (local address 127.0.98.1:52067)
0620 14:30:04.994887 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:04.994899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:04.994964 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:04.995089 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:04.995090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:04.995091 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:04.995190 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.995192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.995880 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.995881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.996851 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:04.996853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.997471 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.997472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:04.997555 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.997559 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:30:04.997886 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 14:30:04.997930 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":12,"thread_start_us":90,"threads_started":1}
I20260620 14:30:04.997980 18240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6763bb626c443d866821d5e67bc8f1" candidate_uuid: "25b52231eaf040adbfae1375946a00f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e23c6372ee844a7b2c3f543ccf016e0" is_pre_election: true
I20260620 14:30:04.998116 18240 raft_consensus.cc:2454] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25b52231eaf040adbfae1375946a00f5 in term 0.
I20260620 14:30:04.998427 18040 leader_election.cc:302] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25b52231eaf040adbfae1375946a00f5, 8e23c6372ee844a7b2c3f543ccf016e0; no voters:
I20260620 14:30:04.998432 18374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6763bb626c443d866821d5e67bc8f1" candidate_uuid: "25b52231eaf040adbfae1375946a00f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31957a5ddd3a4cd3ab282f05726190d9" is_pre_election: true
I20260620 14:30:04.998565 18374 raft_consensus.cc:2454] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25b52231eaf040adbfae1375946a00f5 in term 0.
I20260620 14:30:04.998564 18443 raft_consensus.cc:2781] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:04.998628 18443 raft_consensus.cc:482] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:04.998662 18443 raft_consensus.cc:3037] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:04.999321 18443 raft_consensus.cc:504] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e23c6372ee844a7b2c3f543ccf016e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37721 } } peers { permanent_uuid: "31957a5ddd3a4cd3ab282f05726190d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } } peers { permanent_uuid: "25b52231eaf040adbfae1375946a00f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46707 } }
I20260620 14:30:04.999471 18443 leader_election.cc:288] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [CANDIDATE]: Term 1 election: Requested vote from peers 8e23c6372ee844a7b2c3f543ccf016e0 (127.0.98.2:37721), 31957a5ddd3a4cd3ab282f05726190d9 (127.0.98.3:34523)
I20260620 14:30:04.999601 18240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6763bb626c443d866821d5e67bc8f1" candidate_uuid: "25b52231eaf040adbfae1375946a00f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e23c6372ee844a7b2c3f543ccf016e0"
I20260620 14:30:04.999642 18374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6763bb626c443d866821d5e67bc8f1" candidate_uuid: "25b52231eaf040adbfae1375946a00f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31957a5ddd3a4cd3ab282f05726190d9"
I20260620 14:30:04.999697 18240 raft_consensus.cc:3037] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:04.999711 18374 raft_consensus.cc:3037] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:05.000401 18374 raft_consensus.cc:2454] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25b52231eaf040adbfae1375946a00f5 in term 1.
I20260620 14:30:05.000538 18240 raft_consensus.cc:2454] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25b52231eaf040adbfae1375946a00f5 in term 1.
I20260620 14:30:05.000566 18041 leader_election.cc:302] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25b52231eaf040adbfae1375946a00f5, 31957a5ddd3a4cd3ab282f05726190d9; no voters:
I20260620 14:30:05.000690 18443 raft_consensus.cc:2781] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:05.000837 18443 raft_consensus.cc:686] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 1 LEADER]: Becoming Leader. State: Replica: 25b52231eaf040adbfae1375946a00f5, State: Running, Role: LEADER
I20260620 14:30:05.000928 18443 consensus_queue.cc:237] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e23c6372ee844a7b2c3f543ccf016e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37721 } } peers { permanent_uuid: "31957a5ddd3a4cd3ab282f05726190d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } } peers { permanent_uuid: "25b52231eaf040adbfae1375946a00f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46707 } }
I20260620 14:30:05.001660 17961 catalog_manager.cc:5495] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25b52231eaf040adbfae1375946a00f5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "25b52231eaf040adbfae1375946a00f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e23c6372ee844a7b2c3f543ccf016e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31957a5ddd3a4cd3ab282f05726190d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25b52231eaf040adbfae1375946a00f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46707 } health_report { overall_health: HEALTHY } } }
I20260620 14:30:05.009217 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.006805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.0.98.1:46707)
0620 14:30:05.006847 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:05.006848 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:05.006873 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:30:05.006959 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:05.006960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:05.006980 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:05.007052 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:05.007208 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.007666 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.008433 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.008640 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.008721 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.008862 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:05.008962 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:05.009067 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:05.009097 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 14:30:05.009109 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:30:05.011247 18374 raft_consensus.cc:1261] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9 [term 1 FOLLOWER]: Refusing update from remote peer 25b52231eaf040adbfae1375946a00f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:05.011324 18240 raft_consensus.cc:1261] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0 [term 1 FOLLOWER]: Refusing update from remote peer 25b52231eaf040adbfae1375946a00f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:05.011643 18443 consensus_queue.cc:1035] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31957a5ddd3a4cd3ab282f05726190d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:05.011795 18447 consensus_queue.cc:1035] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e23c6372ee844a7b2c3f543ccf016e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:05.014140 18450 mvcc.cc:205] Tried to move back new op lower bound from 7298931937322979328 to 7298931937284300800. Current Snapshot: MvccSnapshot[applied={T|T < 7298931937322979328}]
I20260620 14:30:05.014639 18451 mvcc.cc:205] Tried to move back new op lower bound from 7298931937322979328 to 7298931937284300800. Current Snapshot: MvccSnapshot[applied={T|T < 7298931937322979328}]
I20260620 14:30:05.016480 17961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:05.016564 17961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:05.017336 17961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 020c63be79e847d2815464f1d7f30d3d: Sending DeleteTablet for 3 replicas of tablet 1b6763bb626c443d866821d5e67bc8f1
I20260620 14:30:05.017737 18086 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b6763bb626c443d866821d5e67bc8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54522
I20260620 14:30:05.017720 18220 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b6763bb626c443d866821d5e67bc8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41578
I20260620 14:30:05.017971 18460 tablet_replica.cc:330] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5: stopping tablet replica
I20260620 14:30:05.018054 18461 tablet_replica.cc:330] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0: stopping tablet replica
I20260620 14:30:05.018079 18460 raft_consensus.cc:2229] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:30:05.018128 18461 raft_consensus.cc:2229] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:05.018097 18354 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b6763bb626c443d866821d5e67bc8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45182
I20260620 14:30:05.018177 18460 raft_consensus.cc:2258] T 1b6763bb626c443d866821d5e67bc8f1 P 25b52231eaf040adbfae1375946a00f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:05.018225 18461 raft_consensus.cc:2258] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:05.018224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 18023
I20260620 14:30:05.019033 18462 tablet_replica.cc:330] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9: stopping tablet replica
I20260620 14:30:05.019142 18462 raft_consensus.cc:2229] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:05.019244 18462 raft_consensus.cc:2258] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:05.019613 18461 ts_tablet_manager.cc:1905] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:05.019666 18462 ts_tablet_manager.cc:1905] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:05.021449 18462 ts_tablet_manager.cc:1918] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:05.021449 18461 ts_tablet_manager.cc:1918] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:05.021528 18461 log.cc:1196] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/1b6763bb626c443d866821d5e67bc8f1
I20260620 14:30:05.021528 18462 log.cc:1196] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/1b6763bb626c443d866821d5e67bc8f1
I20260620 14:30:05.021824 18462 ts_tablet_manager.cc:1939] T 1b6763bb626c443d866821d5e67bc8f1 P 31957a5ddd3a4cd3ab282f05726190d9: Deleting consensus metadata
I20260620 14:30:05.021836 18461 ts_tablet_manager.cc:1939] T 1b6763bb626c443d866821d5e67bc8f1 P 8e23c6372ee844a7b2c3f543ccf016e0: Deleting consensus metadata
I20260620 14:30:05.022289 17947 catalog_manager.cc:4841] TS 31957a5ddd3a4cd3ab282f05726190d9 (127.0.98.3:34523): tablet 1b6763bb626c443d866821d5e67bc8f1 (table test-table [id=7ed4575048f84823b68b0d958de39e1b]) successfully deleted
I20260620 14:30:05.022362 17946 catalog_manager.cc:4841] TS 8e23c6372ee844a7b2c3f543ccf016e0 (127.0.98.2:37721): tablet 1b6763bb626c443d866821d5e67bc8f1 (table test-table [id=7ed4575048f84823b68b0d958de39e1b]) successfully deleted
W20260620 14:30:05.027202 17946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46707 (error 108)
I20260620 14:30:05.027771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 18157
I20260620 14:30:05.028398 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.027660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46707 (local address 127.0.0.1:54550)
0620 14:30:05.028288 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:05.028335 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600}
W20260620 14:30:05.028620 17946 catalog_manager.cc:4584] TS 25b52231eaf040adbfae1375946a00f5 (127.0.98.1:46707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b6763bb626c443d866821d5e67bc8f1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46707: connect: Connection refused (error 111)
I20260620 14:30:05.033625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 18291
I20260620 14:30:05.038214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 17929
2026-06-20T14:30:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1971 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T14:30:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:05Z Disabled control of system clock
I20260620 14:30:05.072780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45161
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:30:05.154649 18467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:05.154881 18467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:05.154903 18467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:05.156116 18467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:05.156169 18467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:05.156183 18467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:05.156201 18467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:05.156214 18467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:05.157598 18467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:05.157874 18467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:05.158201 18467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:05.163152 18475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:05.163203 18473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:05.163178 18472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:05.163580 18467 server_base.cc:1034] running on GCE node
I20260620 14:30:05.163900 18467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:05.164212 18467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:05.165429 18467 hybrid_clock.cc:648] HybridClock initialized: now 1781965805165413 us; error 31 us; skew 500 ppm
I20260620 14:30:05.166911 18467 webserver.cc:460] Webserver started at http://127.0.98.62:46285/ using document root <none> and password file <none>
I20260620 14:30:05.167230 18467 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:05.167309 18467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:05.167464 18467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:05.168591 18467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "f950267e317a4344a57667c546a44ff3"
format_stamp: "Formatted at 2026-06-20 14:30:05 on dist-test-slave-hfc3"
server_key: "4c6bf8fe7b6a171faec734ab3fafd186"
server_key_iv: "5dc07fdf871b50dd52e3c04a249d8ac6"
server_key_version: "encryptionkey@0"
I20260620 14:30:05.168965 18467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "f950267e317a4344a57667c546a44ff3"
format_stamp: "Formatted at 2026-06-20 14:30:05 on dist-test-slave-hfc3"
server_key: "4c6bf8fe7b6a171faec734ab3fafd186"
server_key_iv: "5dc07fdf871b50dd52e3c04a249d8ac6"
server_key_version: "encryptionkey@0"
I20260620 14:30:05.170318 18467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:05.171131 18481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:05.171298 18467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:30:05.171388 18467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "f950267e317a4344a57667c546a44ff3"
format_stamp: "Formatted at 2026-06-20 14:30:05 on dist-test-slave-hfc3"
server_key: "4c6bf8fe7b6a171faec734ab3fafd186"
server_key_iv: "5dc07fdf871b50dd52e3c04a249d8ac6"
server_key_version: "encryptionkey@0"
I20260620 14:30:05.171471 18467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:05.191612 18467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:05.191956 18467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:05.192114 18467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:05.196264 18467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45161
I20260620 14:30:05.196302 18533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45161 every 8 connection(s)
I20260620 14:30:05.196657 18467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:05.197404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 18467
I20260620 14:30:05.197537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:05.197729 18534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:05.197744 392 external_mini_cluster.cc:1428] Setting key 6641d2d451403d3584ed1e811585fbac
I20260620 14:30:05.200860 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3: Bootstrap starting.
I20260620 14:30:05.201553 18534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:05.201857 18534 log.cc:824] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:05.202602 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3: No bootstrap required, opened a new log
I20260620 14:30:05.203728 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.198325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48712 (local address 127.0.98.62:45161)
0620 14:30:05.198622 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:05.198630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:05.198656 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:30:05.199207 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:05.199214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:05.199300 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:05.199425 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:05.199666 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.200539 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.201639 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.202111 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.202274 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.203081 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:05.203086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:05.203086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:05.203279 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:05.203314 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:05.203324 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:05.203404 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:05.203480 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 14:30:05.203563 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":118,"threads_started":1}
I20260620 14:30:05.204603 18534 raft_consensus.cc:348] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f950267e317a4344a57667c546a44ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45161 } }
I20260620 14:30:05.204739 18534 raft_consensus.cc:374] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:05.204762 18534 raft_consensus.cc:729] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f950267e317a4344a57667c546a44ff3, State: Initialized, Role: FOLLOWER
I20260620 14:30:05.204916 18534 consensus_queue.cc:260] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f950267e317a4344a57667c546a44ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45161 } }
I20260620 14:30:05.204980 18534 raft_consensus.cc:388] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:05.205017 18534 raft_consensus.cc:482] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:05.205068 18534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:05.205715 18534 raft_consensus.cc:504] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f950267e317a4344a57667c546a44ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45161 } }
I20260620 14:30:05.205842 18534 leader_election.cc:302] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f950267e317a4344a57667c546a44ff3; no voters:
I20260620 14:30:05.205996 18534 leader_election.cc:288] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:05.206071 18539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:05.206243 18539 raft_consensus.cc:686] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [term 1 LEADER]: Becoming Leader. State: Replica: f950267e317a4344a57667c546a44ff3, State: Running, Role: LEADER
I20260620 14:30:05.206271 18534 sys_catalog.cc:564] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:05.206355 18539 consensus_queue.cc:237] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f950267e317a4344a57667c546a44ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45161 } }
I20260620 14:30:05.206738 18540 sys_catalog.cc:455] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f950267e317a4344a57667c546a44ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f950267e317a4344a57667c546a44ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45161 } } }
I20260620 14:30:05.206950 18540 sys_catalog.cc:458] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:05.206760 18541 sys_catalog.cc:455] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f950267e317a4344a57667c546a44ff3. Latest consensus state: current_term: 1 leader_uuid: "f950267e317a4344a57667c546a44ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f950267e317a4344a57667c546a44ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45161 } } }
I20260620 14:30:05.207074 18541 sys_catalog.cc:458] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:05.207332 18548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:05.207858 18548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:05.209548 18548 catalog_manager.cc:1347] Generated new cluster ID: 14e00242337147f7a89d8525afa6e6b0
I20260620 14:30:05.209614 18548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:05.236721 18548 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:05.237300 18548 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:05.241917 18548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3: Generated new TSK 0
I20260620 14:30:05.242131 18548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:05.252487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45161
--builtin_ntp_servers=127.0.98.20:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:30:05.335829 18558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:05.336045 18558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:05.336067 18558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:05.337224 18558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:05.337273 18558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:05.337325 18558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:30:05.338712 18558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:05.338964 18558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:05.339278 18558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:05.344316 18563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:05.344326 18564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:05.344333 18566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:05.344600 18558 server_base.cc:1034] running on GCE node
I20260620 14:30:05.344827 18558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:05.345088 18558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:05.346267 18558 hybrid_clock.cc:648] HybridClock initialized: now 1781965805346254 us; error 43 us; skew 500 ppm
I20260620 14:30:05.352571 18558 webserver.cc:460] Webserver started at http://127.0.98.1:42507/ using document root <none> and password file <none>
I20260620 14:30:05.352870 18558 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:05.352952 18558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:05.353127 18558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:05.354660 18558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "c8403806574b408088dff6a1cddd572d"
format_stamp: "Formatted at 2026-06-20 14:30:05 on dist-test-slave-hfc3"
server_key: "289d6693cfdfbcc63ea4302107bb693d"
server_key_iv: "544bac8798fb8f98105bfacce9d4262b"
server_key_version: "encryptionkey@0"
I20260620 14:30:05.355058 18558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c8403806574b408088dff6a1cddd572d"
format_stamp: "Formatted at 2026-06-20 14:30:05 on dist-test-slave-hfc3"
server_key: "289d6693cfdfbcc63ea4302107bb693d"
server_key_iv: "544bac8798fb8f98105bfacce9d4262b"
server_key_version: "encryptionkey@0"
I20260620 14:30:05.356415 18558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:30:05.357143 18572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:05.357312 18558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:05.357415 18558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "c8403806574b408088dff6a1cddd572d"
format_stamp: "Formatted at 2026-06-20 14:30:05 on dist-test-slave-hfc3"
server_key: "289d6693cfdfbcc63ea4302107bb693d"
server_key_iv: "544bac8798fb8f98105bfacce9d4262b"
server_key_version: "encryptionkey@0"
I20260620 14:30:05.357503 18558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:05.375391 18558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:05.375732 18558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:05.375891 18558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:05.376135 18558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:05.376492 18558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:05.376549 18558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:05.376592 18558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:05.376621 18558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:05.382282 18558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35263
I20260620 14:30:05.382320 18685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35263 every 8 connection(s)
I20260620 14:30:05.382673 18558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:05.387674 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.383474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60207 (local address 127.0.98.62:45161)
0620 14:30:05.383539 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:05.383541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:05.383778 (+ 237us) server_negotiation.cc:407] Connection header received
0620 14:30:05.384239 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:05.384240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:05.384252 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:05.384321 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:05.384982 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.385482 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.386437 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.386614 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.386665 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.387480 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:05.387480 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:05.387481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:05.387485 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:05.387496 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:05.387499 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:05.387538 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:05.387596 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:30:05.387616 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:30:05.387863 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.383459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45161 (local address 127.0.98.1:60207)
0620 14:30:05.383638 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:05.383680 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:05.384160 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:05.384374 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:05.384376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:05.384384 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:05.384900 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.384904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.385566 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.385567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.386361 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.386363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.386685 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.386685 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.386787 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.387358 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:05.387359 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:05.387387 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:05.387409 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:05.387538 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:05.387540 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:05.387542 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:30:05.387586 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:30:05.387659 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260620 14:30:05.388231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 18558
I20260620 14:30:05.388346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:05.388455 18686 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45161
I20260620 14:30:05.388563 18686 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:05.388592 392 external_mini_cluster.cc:1428] Setting key 02b74cb9e5f596ec148e1a0b2d914317
I20260620 14:30:05.388764 18686 heartbeater.cc:511] Master 127.0.98.62:45161 requested a full tablet report, sending...
I20260620 14:30:05.389317 18497 ts_manager.cc:194] Registered new tserver with Master: c8403806574b408088dff6a1cddd572d (127.0.98.1:35263)
I20260620 14:30:05.389611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45161
--builtin_ntp_servers=127.0.98.20:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:30:05.390085 18497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60207
W20260620 14:30:05.471345 18689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:05.471614 18689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:05.471654 18689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:05.472803 18689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:05.472870 18689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:05.472913 18689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:30:05.474431 18689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:05.474709 18689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:05.475024 18689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:05.480131 18697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:05.480189 18689 server_base.cc:1034] running on GCE node
W20260620 14:30:05.480155 18694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:05.480155 18695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:05.480556 18689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:05.480772 18689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:05.481912 18689 hybrid_clock.cc:648] HybridClock initialized: now 1781965805481900 us; error 28 us; skew 500 ppm
I20260620 14:30:05.483233 18689 webserver.cc:460] Webserver started at http://127.0.98.2:33991/ using document root <none> and password file <none>
I20260620 14:30:05.483477 18689 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:05.483547 18689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:05.483700 18689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:05.484784 18689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "6587fa5fbccc4f4581e35b9533942582"
format_stamp: "Formatted at 2026-06-20 14:30:05 on dist-test-slave-hfc3"
server_key: "6116c75285cb1fcb0b3233d187817896"
server_key_iv: "7a97a4a144538984bfbc2e921efe8451"
server_key_version: "encryptionkey@0"
I20260620 14:30:05.485143 18689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6587fa5fbccc4f4581e35b9533942582"
format_stamp: "Formatted at 2026-06-20 14:30:05 on dist-test-slave-hfc3"
server_key: "6116c75285cb1fcb0b3233d187817896"
server_key_iv: "7a97a4a144538984bfbc2e921efe8451"
server_key_version: "encryptionkey@0"
I20260620 14:30:05.486514 18689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:05.487279 18703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:05.487459 18689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:05.487574 18689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "6587fa5fbccc4f4581e35b9533942582"
format_stamp: "Formatted at 2026-06-20 14:30:05 on dist-test-slave-hfc3"
server_key: "6116c75285cb1fcb0b3233d187817896"
server_key_iv: "7a97a4a144538984bfbc2e921efe8451"
server_key_version: "encryptionkey@0"
I20260620 14:30:05.487663 18689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:05.503110 18689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:05.503437 18689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:05.503593 18689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:05.503829 18689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:05.504166 18689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:05.504220 18689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:05.504264 18689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:05.504293 18689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:05.510576 18689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46653
I20260620 14:30:05.510624 18816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46653 every 8 connection(s)
I20260620 14:30:05.510967 18689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:05.514693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 18689
I20260620 14:30:05.514813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:05.515020 392 external_mini_cluster.cc:1428] Setting key 4b3ced78afe135e1211819fbadab52bc
I20260620 14:30:05.516270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45161
--builtin_ntp_servers=127.0.98.20:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:30:05.518302 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.511729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42447 (local address 127.0.98.62:45161)
0620 14:30:05.511776 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:05.511779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:05.511988 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:30:05.512524 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:05.512526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:05.512538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:05.512592 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:05.513318 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.513820 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.515151 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.515341 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.515434 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.517939 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:05.517941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:05.517941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:05.517947 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:05.517969 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:05.517974 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:05.518042 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:05.518167 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:30:05.518198 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:30:05.518522 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.511685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45161 (local address 127.0.98.2:42447)
0620 14:30:05.511864 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:05.511899 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:05.512415 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:05.512683 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:05.512693 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:05.512702 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:05.513210 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.513215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.513919 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.513921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.515067 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.515070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.515452 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.515453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.515568 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.517785 (+ 2217us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:05.517785 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:05.517816 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:05.517840 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:05.518070 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:05.518077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:05.518080 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:05.518152 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:30:05.518242 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260620 14:30:05.519034 18817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45161
I20260620 14:30:05.519129 18817 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:05.519335 18817 heartbeater.cc:511] Master 127.0.98.62:45161 requested a full tablet report, sending...
I20260620 14:30:05.519726 18497 ts_manager.cc:194] Registered new tserver with Master: 6587fa5fbccc4f4581e35b9533942582 (127.0.98.2:46653)
I20260620 14:30:05.520282 18497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42447
W20260620 14:30:05.599134 18820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:05.599360 18820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:05.599382 18820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:05.600448 18820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:05.600479 18820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:05.600503 18820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:30:05.601758 18820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:05.601939 18820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:05.602182 18820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:05.607131 18826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:05.607131 18825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:05.607116 18828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:05.607594 18820 server_base.cc:1034] running on GCE node
I20260620 14:30:05.607863 18820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:05.608114 18820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:05.609283 18820 hybrid_clock.cc:648] HybridClock initialized: now 1781965805609273 us; error 40 us; skew 500 ppm
I20260620 14:30:05.610620 18820 webserver.cc:460] Webserver started at http://127.0.98.3:33611/ using document root <none> and password file <none>
I20260620 14:30:05.610966 18820 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:05.611037 18820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:05.611182 18820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:05.612963 18820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "f5abc2747c584c6387860a7c3c1beddf"
format_stamp: "Formatted at 2026-06-20 14:30:05 on dist-test-slave-hfc3"
server_key: "695d5f02e57a81fc47a9028438a3a5ee"
server_key_iv: "bd587283de5642bdaa838cf634c52dc9"
server_key_version: "encryptionkey@0"
I20260620 14:30:05.613409 18820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f5abc2747c584c6387860a7c3c1beddf"
format_stamp: "Formatted at 2026-06-20 14:30:05 on dist-test-slave-hfc3"
server_key: "695d5f02e57a81fc47a9028438a3a5ee"
server_key_iv: "bd587283de5642bdaa838cf634c52dc9"
server_key_version: "encryptionkey@0"
I20260620 14:30:05.616147 18820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:30:05.617396 18834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:05.617699 18820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:05.617853 18820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "f5abc2747c584c6387860a7c3c1beddf"
format_stamp: "Formatted at 2026-06-20 14:30:05 on dist-test-slave-hfc3"
server_key: "695d5f02e57a81fc47a9028438a3a5ee"
server_key_iv: "bd587283de5642bdaa838cf634c52dc9"
server_key_version: "encryptionkey@0"
I20260620 14:30:05.617933 18820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:05.653867 18820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:05.654224 18820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:05.654408 18820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:05.654668 18820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:05.655032 18820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:05.655089 18820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:05.655134 18820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:05.655160 18820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:05.661139 18820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34819
I20260620 14:30:05.661221 18947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34819 every 8 connection(s)
I20260620 14:30:05.661581 18820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:05.662055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 18820
I20260620 14:30:05.662163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:05.662413 392 external_mini_cluster.cc:1428] Setting key 43777528cf50abd66d8328ae12898fc4
I20260620 14:30:05.666965 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.662424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54379 (local address 127.0.98.62:45161)
0620 14:30:05.662470 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:05.662472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:05.662759 (+ 287us) server_negotiation.cc:407] Connection header received
0620 14:30:05.663448 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:05.663449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:05.663459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:05.663517 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:05.664188 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.664693 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.665675 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.665869 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.665926 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.666749 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:05.666750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:05.666751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:05.666756 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:05.666773 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:05.666777 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:05.666820 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:05.666881 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:30:05.666902 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:30:05.667203 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.662433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45161 (local address 127.0.98.3:54379)
0620 14:30:05.662642 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:05.662691 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:05.663348 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:05.663577 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:05.663581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:05.663589 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:05.664113 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.664117 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.664777 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.664778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.665590 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.665592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.665945 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.665946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.666055 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.666619 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:05.666620 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:05.666649 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:05.666672 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:05.666820 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:05.666822 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:05.666824 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:30:05.666872 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:30:05.666950 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260620 14:30:05.667709 18948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45161
I20260620 14:30:05.667789 18948 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:05.667996 18948 heartbeater.cc:511] Master 127.0.98.62:45161 requested a full tablet report, sending...
I20260620 14:30:05.668378 18497 ts_manager.cc:194] Registered new tserver with Master: f5abc2747c584c6387860a7c3c1beddf (127.0.98.3:34819)
I20260620 14:30:05.668709 18497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54379
I20260620 14:30:05.674219 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:30:05.678459 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.675754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.0.98.62:45161)
0620 14:30:05.675799 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:05.675806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:05.675904 (+ 98us) server_negotiation.cc:407] Connection header received
0620 14:30:05.675945 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:05.675946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:05.675954 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:05.676002 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:05.676134 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.676600 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.677314 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.677487 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.677536 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.678156 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:05.678157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:05.678157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:05.678161 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:05.678174 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:05.678178 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:05.678255 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:05.678323 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:30:05.678375 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:30:05.680249 18497 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:05.681866 18497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:05.688398 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.684274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.0.98.1:35263)
0620 14:30:05.684560 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:05.684566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:05.684680 (+ 114us) server_negotiation.cc:407] Connection header received
0620 14:30:05.684759 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:05.684770 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:05.684786 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:05.684887 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:05.685065 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.685936 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.687361 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.687989 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.688083 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.688180 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:05.688271 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 14:30:05.688308 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":128,"threads_started":1}
I20260620 14:30:05.688386 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.684171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35263 (local address 127.0.0.1:50812)
0620 14:30:05.684616 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:05.684627 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:05.684692 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:05.684900 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:05.684903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:05.684905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:05.684980 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.684982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.686017 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.686019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.687268 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.687270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.688087 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.688088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.688164 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.688177 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:30:05.688252 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:30:05.688307 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260620 14:30:05.688614 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.684171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46653 (local address 127.0.0.1:48708)
0620 14:30:05.684514 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:05.684559 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:05.684627 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:05.685083 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:05.685083 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:05.685085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:05.685119 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.685120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.686407 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.686408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.687432 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.687433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.688366 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.688367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.688429 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.688434 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:30:05.688466 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 14:30:05.688513 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":70,"threads_started":1}
I20260620 14:30:05.688908 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.684381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34819 (local address 127.0.0.1:54388)
0620 14:30:05.684727 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:05.684733 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:05.684810 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:05.685326 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:05.685327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:05.685328 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:05.685371 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.685372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.686588 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.686589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.687601 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.687603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.688658 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.688659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.688721 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.688725 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:30:05.688775 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:30:05.688813 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":76,"threads_started":1}
I20260620 14:30:05.688972 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.684384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.0.98.2:46653)
0620 14:30:05.684939 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:05.684943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:05.684953 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:30:05.684992 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:05.684994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:05.685010 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:05.685088 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:05.685177 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.686292 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.687514 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.688259 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.688733 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.688830 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:05.688853 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 14:30:05.688887 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":86,"threads_started":1}
I20260620 14:30:05.689102 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.684470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.0.98.3:34819)
0620 14:30:05.684641 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:05.684646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:05.684817 (+ 171us) server_negotiation.cc:407] Connection header received
0620 14:30:05.685193 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:05.685199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:05.685221 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:05.685313 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:05.685439 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.686511 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.687695 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.688406 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.688642 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.688813 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:05.688843 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 14:30:05.688883 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260620 14:30:05.689554 18751 tablet_service.cc:1467] Processing CreateTablet for tablet 36a2cba6518d4a9f88e3ceb4955f8b56 (DEFAULT_TABLE table=test-table [id=ce4b2a1f31d94289b2be4faa77ea4165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:05.689530 18620 tablet_service.cc:1467] Processing CreateTablet for tablet 36a2cba6518d4a9f88e3ceb4955f8b56 (DEFAULT_TABLE table=test-table [id=ce4b2a1f31d94289b2be4faa77ea4165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:05.689922 18751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a2cba6518d4a9f88e3ceb4955f8b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:05.689932 18620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a2cba6518d4a9f88e3ceb4955f8b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:05.689888 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 36a2cba6518d4a9f88e3ceb4955f8b56 (DEFAULT_TABLE table=test-table [id=ce4b2a1f31d94289b2be4faa77ea4165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:05.690191 18882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a2cba6518d4a9f88e3ceb4955f8b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:05.692672 18962 tablet_bootstrap.cc:492] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582: Bootstrap starting.
I20260620 14:30:05.693383 18964 tablet_bootstrap.cc:492] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf: Bootstrap starting.
I20260620 14:30:05.693531 18962 tablet_bootstrap.cc:654] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:05.693885 18962 log.cc:824] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:05.694052 18964 tablet_bootstrap.cc:654] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:05.694413 18964 log.cc:824] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:05.694514 18963 tablet_bootstrap.cc:492] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d: Bootstrap starting.
I20260620 14:30:05.694792 18962 tablet_bootstrap.cc:492] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582: No bootstrap required, opened a new log
I20260620 14:30:05.694873 18962 ts_tablet_manager.cc:1397] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:05.695132 18963 tablet_bootstrap.cc:654] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:05.695163 18964 tablet_bootstrap.cc:492] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf: No bootstrap required, opened a new log
I20260620 14:30:05.695216 18964 ts_tablet_manager.cc:1397] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:30:05.695452 18963 log.cc:824] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:05.696262 18963 tablet_bootstrap.cc:492] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d: No bootstrap required, opened a new log
I20260620 14:30:05.696332 18963 ts_tablet_manager.cc:1397] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:05.697198 18962 raft_consensus.cc:348] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8403806574b408088dff6a1cddd572d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "f5abc2747c584c6387860a7c3c1beddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "6587fa5fbccc4f4581e35b9533942582" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } }
I20260620 14:30:05.697361 18962 raft_consensus.cc:374] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:05.697404 18962 raft_consensus.cc:729] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6587fa5fbccc4f4581e35b9533942582, State: Initialized, Role: FOLLOWER
I20260620 14:30:05.697542 18962 consensus_queue.cc:260] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8403806574b408088dff6a1cddd572d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "f5abc2747c584c6387860a7c3c1beddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "6587fa5fbccc4f4581e35b9533942582" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } }
I20260620 14:30:05.697827 18817 heartbeater.cc:503] Master 127.0.98.62:45161 was elected leader, sending a full tablet report...
I20260620 14:30:05.697880 18964 raft_consensus.cc:348] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8403806574b408088dff6a1cddd572d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "f5abc2747c584c6387860a7c3c1beddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "6587fa5fbccc4f4581e35b9533942582" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } }
I20260620 14:30:05.698036 18964 raft_consensus.cc:374] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:05.698071 18964 raft_consensus.cc:729] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5abc2747c584c6387860a7c3c1beddf, State: Initialized, Role: FOLLOWER
I20260620 14:30:05.698207 18964 consensus_queue.cc:260] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8403806574b408088dff6a1cddd572d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "f5abc2747c584c6387860a7c3c1beddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "6587fa5fbccc4f4581e35b9533942582" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } }
I20260620 14:30:05.698501 18964 ts_tablet_manager.cc:1428] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 14:30:05.698539 18948 heartbeater.cc:503] Master 127.0.98.62:45161 was elected leader, sending a full tablet report...
I20260620 14:30:05.698084 18962 ts_tablet_manager.cc:1428] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:05.698985 18963 raft_consensus.cc:348] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8403806574b408088dff6a1cddd572d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "f5abc2747c584c6387860a7c3c1beddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "6587fa5fbccc4f4581e35b9533942582" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } }
I20260620 14:30:05.699148 18963 raft_consensus.cc:374] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:05.699199 18963 raft_consensus.cc:729] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8403806574b408088dff6a1cddd572d, State: Initialized, Role: FOLLOWER
I20260620 14:30:05.699318 18963 consensus_queue.cc:260] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8403806574b408088dff6a1cddd572d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "f5abc2747c584c6387860a7c3c1beddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "6587fa5fbccc4f4581e35b9533942582" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } }
I20260620 14:30:05.699604 18963 ts_tablet_manager.cc:1428] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 14:30:05.699636 18686 heartbeater.cc:503] Master 127.0.98.62:45161 was elected leader, sending a full tablet report...
I20260620 14:30:05.735145 18969 raft_consensus.cc:482] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:05.735376 18969 raft_consensus.cc:504] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8403806574b408088dff6a1cddd572d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "f5abc2747c584c6387860a7c3c1beddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "6587fa5fbccc4f4581e35b9533942582" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } }
I20260620 14:30:05.735810 18969 leader_election.cc:288] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8403806574b408088dff6a1cddd572d (127.0.98.1:35263), 6587fa5fbccc4f4581e35b9533942582 (127.0.98.2:46653)
I20260620 14:30:05.738922 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.736194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49301 (local address 127.0.98.1:35263)
0620 14:30:05.736256 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:05.736258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:05.736267 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:30:05.736297 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:05.736298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:05.736313 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:05.736373 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:05.736569 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.737118 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.738118 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.738509 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.738564 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.738571 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:05.738797 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 14:30:05.738847 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:30:05.738986 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.736097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40217 (local address 127.0.98.2:46653)
0620 14:30:05.736141 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:05.736143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:05.736388 (+ 245us) server_negotiation.cc:407] Connection header received
0620 14:30:05.736421 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:05.736422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:05.736435 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:05.736504 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:05.736696 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.737190 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.738152 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.738621 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.738693 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.738705 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:05.738883 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:30:05.738925 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:30:05.738976 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.735999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35263 (local address 127.0.98.3:49301)
0620 14:30:05.736071 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:05.736084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:05.736145 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:05.736408 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:05.736410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:05.736412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:05.736486 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.736488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.737212 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.737213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.738015 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.738017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.738592 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.738593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.738720 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.738724 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:30:05.738776 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:30:05.738883 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":2}
I20260620 14:30:05.739012 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.736003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46653 (local address 127.0.98.3:40217)
0620 14:30:05.736302 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:05.736311 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:05.736362 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:05.736518 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:05.736535 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:05.736537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:05.736614 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.736616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.737284 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.737284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.738024 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:05.738026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.738709 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.738710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:05.738797 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.738802 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:30:05.738858 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:30:05.738895 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":76,"thread_start_us":145,"threads_started":1}
I20260620 14:30:05.739359 18771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a2cba6518d4a9f88e3ceb4955f8b56" candidate_uuid: "f5abc2747c584c6387860a7c3c1beddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6587fa5fbccc4f4581e35b9533942582" is_pre_election: true
I20260620 14:30:05.739372 18640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a2cba6518d4a9f88e3ceb4955f8b56" candidate_uuid: "f5abc2747c584c6387860a7c3c1beddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8403806574b408088dff6a1cddd572d" is_pre_election: true
I20260620 14:30:05.739531 18640 raft_consensus.cc:2454] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5abc2747c584c6387860a7c3c1beddf in term 0.
I20260620 14:30:05.739531 18771 raft_consensus.cc:2454] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5abc2747c584c6387860a7c3c1beddf in term 0.
I20260620 14:30:05.739794 18835 leader_election.cc:302] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6587fa5fbccc4f4581e35b9533942582, f5abc2747c584c6387860a7c3c1beddf; no voters:
I20260620 14:30:05.739933 18969 raft_consensus.cc:2781] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:05.740003 18969 raft_consensus.cc:482] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:05.740021 18969 raft_consensus.cc:3037] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:05.740775 18969 raft_consensus.cc:504] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8403806574b408088dff6a1cddd572d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "f5abc2747c584c6387860a7c3c1beddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "6587fa5fbccc4f4581e35b9533942582" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } }
I20260620 14:30:05.740947 18969 leader_election.cc:288] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [CANDIDATE]: Term 1 election: Requested vote from peers c8403806574b408088dff6a1cddd572d (127.0.98.1:35263), 6587fa5fbccc4f4581e35b9533942582 (127.0.98.2:46653)
I20260620 14:30:05.741153 18771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a2cba6518d4a9f88e3ceb4955f8b56" candidate_uuid: "f5abc2747c584c6387860a7c3c1beddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6587fa5fbccc4f4581e35b9533942582"
I20260620 14:30:05.741178 18640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a2cba6518d4a9f88e3ceb4955f8b56" candidate_uuid: "f5abc2747c584c6387860a7c3c1beddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8403806574b408088dff6a1cddd572d"
I20260620 14:30:05.741236 18771 raft_consensus.cc:3037] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:05.741256 18640 raft_consensus.cc:3037] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:05.742177 18771 raft_consensus.cc:2454] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5abc2747c584c6387860a7c3c1beddf in term 1.
I20260620 14:30:05.742177 18640 raft_consensus.cc:2454] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5abc2747c584c6387860a7c3c1beddf in term 1.
I20260620 14:30:05.742424 18837 leader_election.cc:302] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8403806574b408088dff6a1cddd572d, f5abc2747c584c6387860a7c3c1beddf; no voters:
I20260620 14:30:05.742542 18969 raft_consensus.cc:2781] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:05.742725 18969 raft_consensus.cc:686] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 1 LEADER]: Becoming Leader. State: Replica: f5abc2747c584c6387860a7c3c1beddf, State: Running, Role: LEADER
I20260620 14:30:05.742843 18969 consensus_queue.cc:237] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8403806574b408088dff6a1cddd572d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "f5abc2747c584c6387860a7c3c1beddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "6587fa5fbccc4f4581e35b9533942582" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } }
I20260620 14:30:05.743739 18498 catalog_manager.cc:5495] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf reported cstate change: term changed from 0 to 1, leader changed from <none> to f5abc2747c584c6387860a7c3c1beddf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f5abc2747c584c6387860a7c3c1beddf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8403806574b408088dff6a1cddd572d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5abc2747c584c6387860a7c3c1beddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6587fa5fbccc4f4581e35b9533942582" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } health_report { overall_health: UNKNOWN } } }
I20260620 14:30:05.751736 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.749543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.0.98.3:34819)
0620 14:30:05.749609 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:05.749611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:05.749622 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:30:05.749658 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:05.749659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:05.749677 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:05.749740 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:05.749851 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.750308 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.751090 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:05.751278 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:05.751330 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:05.751449 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:05.751535 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:05.751603 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:05.751646 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:30:05.751654 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:30:05.753594 18771 raft_consensus.cc:1261] T 36a2cba6518d4a9f88e3ceb4955f8b56 P 6587fa5fbccc4f4581e35b9533942582 [term 1 FOLLOWER]: Refusing update from remote peer f5abc2747c584c6387860a7c3c1beddf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:05.753616 18640 raft_consensus.cc:1261] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d [term 1 FOLLOWER]: Refusing update from remote peer f5abc2747c584c6387860a7c3c1beddf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:05.754097 18969 consensus_queue.cc:1035] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8403806574b408088dff6a1cddd572d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:05.754228 18972 consensus_queue.cc:1035] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6587fa5fbccc4f4581e35b9533942582" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:05.756898 18977 mvcc.cc:205] Tried to move back new op lower bound from 7298931940364062720 to 7298931940323557376. Current Snapshot: MvccSnapshot[applied={T|T < 7298931940364062720}]
I20260620 14:30:05.757059 18975 mvcc.cc:205] Tried to move back new op lower bound from 7298931940364062720 to 7298931940323557376. Current Snapshot: MvccSnapshot[applied={T|T < 7298931940364062720 or (T in {7298931940364062720})}]
I20260620 14:30:05.759207 18497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:05.759308 18497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:05.760087 18497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f950267e317a4344a57667c546a44ff3: Sending DeleteTablet for 3 replicas of tablet 36a2cba6518d4a9f88e3ceb4955f8b56
I20260620 14:30:05.760421 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a2cba6518d4a9f88e3ceb4955f8b56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:05 UTC) from {username='slave'} at 127.0.0.1:54388
I20260620 14:30:05.760416 18620 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a2cba6518d4a9f88e3ceb4955f8b56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:05 UTC) from {username='slave'} at 127.0.0.1:50812
I20260620 14:30:05.760524 18751 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a2cba6518d4a9f88e3ceb4955f8b56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:05 UTC) from {username='slave'} at 127.0.0.1:48708
I20260620 14:30:05.760624 18986 tablet_replica.cc:330] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d: stopping tablet replica
I20260620 14:30:05.760643 18985 tablet_replica.cc:330] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf: stopping tablet replica
I20260620 14:30:05.760713 18986 raft_consensus.cc:2229] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:05.760739 18985 raft_consensus.cc:2229] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:30:05.760828 18986 raft_consensus.cc:2258] T 36a2cba6518d4a9f88e3ceb4955f8b56 P c8403806574b408088dff6a1cddd572d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:05.760831 18985 raft_consensus.cc:2258] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:05.761118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 18558
I20260620 14:30:05.763437 18985 ts_tablet_manager.cc:1905] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:05.765555 18985 ts_tablet_manager.cc:1918] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:05.765641 18985 log.cc:1196] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/36a2cba6518d4a9f88e3ceb4955f8b56
I20260620 14:30:05.765904 18985 ts_tablet_manager.cc:1939] T 36a2cba6518d4a9f88e3ceb4955f8b56 P f5abc2747c584c6387860a7c3c1beddf: Deleting consensus metadata
I20260620 14:30:05.766371 18484 catalog_manager.cc:4841] TS f5abc2747c584c6387860a7c3c1beddf (127.0.98.3:34819): tablet 36a2cba6518d4a9f88e3ceb4955f8b56 (table test-table [id=ce4b2a1f31d94289b2be4faa77ea4165]) successfully deleted
W20260620 14:30:05.768381 18484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35263 (error 108)
I20260620 14:30:05.768916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 18689
I20260620 14:30:05.769622 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.769050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35263 (local address 127.0.0.1:50828)
0620 14:30:05.769533 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:05.769585 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459}
W20260620 14:30:05.769760 18484 catalog_manager.cc:4584] TS c8403806574b408088dff6a1cddd572d (127.0.98.1:35263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36a2cba6518d4a9f88e3ceb4955f8b56: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35263: connect: Connection refused (error 111)
I20260620 14:30:05.773727 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:05.773610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46653 (local address 127.0.0.1:48716)
0620 14:30:05.773658 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:05.773684 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 14:30:05.774626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 18820
I20260620 14:30:05.779778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 18467
2026-06-20T14:30:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (735 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:05 dist-test-slave-hfc3 krb5kdc[18991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:05 dist-test-slave-hfc3 krb5kdc[18991](info): set up 2 sockets
Jun 20 14:30:05 dist-test-slave-hfc3 krb5kdc[18991](info): commencing operation
krb5kdc: starting...
W20260620 14:30:06.838157 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:06 dist-test-slave-hfc3 krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:06.957525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41565
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:07.040377 19004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:07.040612 19004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:07.040638 19004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:07.041817 19004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:07.041862 19004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:07.041877 19004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:07.041890 19004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:07.041903 19004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:07.043593 19004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:07.043911 19004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:07.044173 19004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:07.050258 19009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.050207 19010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.050199 19012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:07.050421 19004 server_base.cc:1034] running on GCE node
I20260620 14:30:07.050890 19004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:07.051203 19004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:07.052350 19004 hybrid_clock.cc:648] HybridClock initialized: now 1781965807052327 us; error 41 us; skew 500 ppm
Jun 20 14:30:07 dist-test-slave-hfc3 krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:07.055150 19004 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:07.055723 19004 webserver.cc:460] Webserver started at http://127.0.98.62:39703/ using document root <none> and password file <none>
I20260620 14:30:07.055982 19004 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:07.056058 19004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:07.056216 19004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:07.057343 19004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "92558fc022dd49c982b6f1dffb703a4e"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-hfc3"
server_key: "5cfb7fe1aaab67f10c63e9ff6bee6ad7"
server_key_iv: "523bf1a25d41cdd29c1019c1737ea495"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.057727 19004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "92558fc022dd49c982b6f1dffb703a4e"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-hfc3"
server_key: "5cfb7fe1aaab67f10c63e9ff6bee6ad7"
server_key_iv: "523bf1a25d41cdd29c1019c1737ea495"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.059366 19004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:30:07.060245 19019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.060469 19004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:30:07.060583 19004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "92558fc022dd49c982b6f1dffb703a4e"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-hfc3"
server_key: "5cfb7fe1aaab67f10c63e9ff6bee6ad7"
server_key_iv: "523bf1a25d41cdd29c1019c1737ea495"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.060672 19004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:07.087146 19004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:07.089628 19004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:07.089811 19004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:07.094400 19004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41565
I20260620 14:30:07.094453 19071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41565 every 8 connection(s)
I20260620 14:30:07.094843 19004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:07.095744 19072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:07.098127 19072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e: Bootstrap starting.
I20260620 14:30:07.098842 19072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:07.099179 19072 log.cc:824] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:07.099943 19072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e: No bootstrap required, opened a new log
I20260620 14:30:07.101856 19072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92558fc022dd49c982b6f1dffb703a4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41565 } }
I20260620 14:30:07.102005 19072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:07.102027 19072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92558fc022dd49c982b6f1dffb703a4e, State: Initialized, Role: FOLLOWER
I20260620 14:30:07.102294 19072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92558fc022dd49c982b6f1dffb703a4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41565 } }
I20260620 14:30:07.102440 19072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:07.102483 19072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:07.102535 19072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:07.103048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 19004
I20260620 14:30:07.103173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:07.103314 19072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92558fc022dd49c982b6f1dffb703a4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41565 } }
I20260620 14:30:07.103399 392 external_mini_cluster.cc:1428] Setting key 76d155cb80814ddb2649c3d541c440fd
I20260620 14:30:07.103425 19072 leader_election.cc:302] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92558fc022dd49c982b6f1dffb703a4e; no voters:
I20260620 14:30:07.103605 19072 leader_election.cc:288] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:07.103636 19075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:07.103767 19075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [term 1 LEADER]: Becoming Leader. State: Replica: 92558fc022dd49c982b6f1dffb703a4e, State: Running, Role: LEADER
I20260620 14:30:07.103852 19075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92558fc022dd49c982b6f1dffb703a4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41565 } }
I20260620 14:30:07.103886 19072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:07.104278 19076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92558fc022dd49c982b6f1dffb703a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92558fc022dd49c982b6f1dffb703a4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41565 } } }
I20260620 14:30:07.104413 19077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92558fc022dd49c982b6f1dffb703a4e. Latest consensus state: current_term: 1 leader_uuid: "92558fc022dd49c982b6f1dffb703a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92558fc022dd49c982b6f1dffb703a4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41565 } } }
I20260620 14:30:07.104504 19077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:07.104729 19076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:07.104744 19085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:07.105628 19085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:07.107542 19085 catalog_manager.cc:1347] Generated new cluster ID: 3dc76ec051bd44a0ae56367559cb86e0
I20260620 14:30:07.107602 19085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:30:07 dist-test-slave-hfc3 krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:07.114701 19085 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:07.115391 19085 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:07.115751 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.104126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.0.98.62:41565)
0620 14:30:07.104278 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.104288 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.105487 (+ 1199us) server_negotiation.cc:407] Connection header received
0620 14:30:07.105705 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.105713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.105774 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.105936 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.106764 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.108008 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.108784 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.109237 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.109383 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.111890 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.111894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.111895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.111906 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.113972 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.114464 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.114467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.114467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.114605 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.114862 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.114863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.114864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.115103 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.115278 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.115355 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.115515 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
I20260620 14:30:07.137006 19085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e: Generated new TSK 0
I20260620 14:30:07.137262 19085 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:30:07.168679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41565
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:07.266095 19098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:07.266346 19098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:07.266415 19098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:07.267632 19098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:07.267699 19098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:07.267761 19098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:30:07.269292 19098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:07.269562 19098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:07.269829 19098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:07.274952 19103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.275013 19106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.274996 19104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:07.275373 19098 server_base.cc:1034] running on GCE node
I20260620 14:30:07.275578 19098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:07.275781 19098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:07.276947 19098 hybrid_clock.cc:648] HybridClock initialized: now 1781965807276937 us; error 30 us; skew 500 ppm
Jun 20 14:30:07 dist-test-slave-hfc3 krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:07.279593 19098 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:30:07.279979 19098 webserver.cc:460] Webserver started at http://127.0.98.1:34081/ using document root <none> and password file <none>
I20260620 14:30:07.280158 19098 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:07.280198 19098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:07.280294 19098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:07.281296 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "fafac1af20614c98ac908ab34ddcdd8f"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-hfc3"
server_key: "fe598f77a28afbe7d1627d4f3d543e1a"
server_key_iv: "368f433af265e92366e0b0e352e937c7"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.281594 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fafac1af20614c98ac908ab34ddcdd8f"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-hfc3"
server_key: "fe598f77a28afbe7d1627d4f3d543e1a"
server_key_iv: "368f433af265e92366e0b0e352e937c7"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.282963 19098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:30:07.283660 19113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.283845 19098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:30:07.283918 19098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "fafac1af20614c98ac908ab34ddcdd8f"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-hfc3"
server_key: "fe598f77a28afbe7d1627d4f3d543e1a"
server_key_iv: "368f433af265e92366e0b0e352e937c7"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.283972 19098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:07.295650 19098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:07.297932 19098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:07.298044 19098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:07.298221 19098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:07.298580 19098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:07.298642 19098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.298677 19098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:07.298709 19098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.304692 19098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41195
I20260620 14:30:07.304775 19226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41195 every 8 connection(s)
I20260620 14:30:07.305190 19098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:30:07 dist-test-slave-hfc3 krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:07.315017 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.306093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59111 (local address 127.0.98.62:41565)
0620 14:30:07.306138 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.306140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.306411 (+ 271us) server_negotiation.cc:407] Connection header received
0620 14:30:07.307392 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.307393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.307406 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.307472 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.308185 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.308700 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.309748 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.309951 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.310036 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.312365 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.312367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.312367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.312374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.313583 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.313987 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.313989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.313989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.314054 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.314293 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.314294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.314294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.314477 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.314535 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.314827 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.314942 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:30:07.315122 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.306065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41565 (local address 127.0.98.1:59111)
0620 14:30:07.306270 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.306306 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.307273 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.307545 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.307549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.307879 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:07.308074 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.308078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.308807 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.308808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.309633 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.309635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.310048 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.310049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.310175 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.310763 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:07.310771 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:07.312254 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:07.313655 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.313661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.313663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.313892 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.314134 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.314135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.314135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.314234 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.314536 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:07.314541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:07.314655 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.314784 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.314929 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
I20260620 14:30:07.315330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 19098
I20260620 14:30:07.315439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:07.315668 392 external_mini_cluster.cc:1428] Setting key d473a55d88a0d1cdfb485765177e1430
I20260620 14:30:07.316409 19227 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41565
I20260620 14:30:07.316560 19227 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:07.316769 19227 heartbeater.cc:511] Master 127.0.98.62:41565 requested a full tablet report, sending...
I20260620 14:30:07.317329 19036 ts_manager.cc:194] Registered new tserver with Master: fafac1af20614c98ac908ab34ddcdd8f (127.0.98.1:41195)
I20260620 14:30:07.318053 19036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59111
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:30:07.339248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41565
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:07.421160 19232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:07.421371 19232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:07.421396 19232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:07.422581 19232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:07.422626 19232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:07.422654 19232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:30:07.424165 19232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:07.424413 19232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:07.424661 19232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:07.429657 19240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.429713 19238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.429677 19237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:07.430171 19232 server_base.cc:1034] running on GCE node
I20260620 14:30:07.430461 19232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:07.430732 19232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:07.431886 19232 hybrid_clock.cc:648] HybridClock initialized: now 1781965807431879 us; error 33 us; skew 500 ppm
Jun 20 14:30:07 dist-test-slave-hfc3 krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:07.434609 19232 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:30:07.435158 19232 webserver.cc:460] Webserver started at http://127.0.98.2:37717/ using document root <none> and password file <none>
I20260620 14:30:07.435420 19232 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:07.435493 19232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:07.435644 19232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:07.436744 19232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "bf60f1dcc18a43dda0ff0aba25635ab9"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-hfc3"
server_key: "6f6afe2de61cc1d3feec883e2a938c4b"
server_key_iv: "6bd636462b4d53ff8f3b84adc2340035"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.437131 19232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bf60f1dcc18a43dda0ff0aba25635ab9"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-hfc3"
server_key: "6f6afe2de61cc1d3feec883e2a938c4b"
server_key_iv: "6bd636462b4d53ff8f3b84adc2340035"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.438565 19232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:07.439375 19247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.439539 19232 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:07.439649 19232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "bf60f1dcc18a43dda0ff0aba25635ab9"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-hfc3"
server_key: "6f6afe2de61cc1d3feec883e2a938c4b"
server_key_iv: "6bd636462b4d53ff8f3b84adc2340035"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.439731 19232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:07.455662 19232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:07.457908 19232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:07.458089 19232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:07.458393 19232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:07.458736 19232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:07.458798 19232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.458841 19232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:07.458868 19232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.465190 19232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41657
I20260620 14:30:07.465240 19360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41657 every 8 connection(s)
I20260620 14:30:07.465577 19232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:30:07 dist-test-slave-hfc3 krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:07.474848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 19232
I20260620 14:30:07.474980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:07.475185 392 external_mini_cluster.cc:1428] Setting key 4540d407cc36ebf9d4c6a21400b9a661
I20260620 14:30:07.475299 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.466365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52987 (local address 127.0.98.62:41565)
0620 14:30:07.466452 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.466454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.466583 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:30:07.467577 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.467578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.467591 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.467630 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.468301 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.468813 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.469861 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.470061 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.470168 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.472494 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.472496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.472497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.472504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.473789 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.474208 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.474210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.474210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.474277 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.474566 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.474567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.474567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.474733 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.474809 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.475113 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.475203 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:30:07.475507 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.466267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41565 (local address 127.0.98.2:52987)
0620 14:30:07.466435 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.466487 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.467457 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.467710 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.467714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.467996 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:07.468188 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.468192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.468915 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.468917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.469761 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.469763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.470177 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.470178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.470320 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.470900 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:07.470909 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:07.472395 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:07.473890 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.473897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.473899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.474126 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.474377 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.474377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.474378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.474489 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.474810 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:07.474816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:07.474938 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.475070 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.475206 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260620 14:30:07.476032 19361 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41565
I20260620 14:30:07.476145 19361 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:07.476356 19361 heartbeater.cc:511] Master 127.0.98.62:41565 requested a full tablet report, sending...
I20260620 14:30:07.476716 19036 ts_manager.cc:194] Registered new tserver with Master: bf60f1dcc18a43dda0ff0aba25635ab9 (127.0.98.2:41657)
I20260620 14:30:07.477097 19036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52987
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:30:07.500829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41565
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:07.584270 19366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:07.584530 19366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:07.584573 19366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:07.585783 19366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:07.585856 19366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:07.585904 19366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:30:07.587550 19366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:07.587828 19366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:07.588063 19366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:07.593452 19371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.593452 19372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.593467 19374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:07.593528 19366 server_base.cc:1034] running on GCE node
I20260620 14:30:07.593948 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:07.594200 19366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:07.595357 19366 hybrid_clock.cc:648] HybridClock initialized: now 1781965807595326 us; error 47 us; skew 500 ppm
Jun 20 14:30:07 dist-test-slave-hfc3 krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:07.597869 19366 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:30:07.598408 19366 webserver.cc:460] Webserver started at http://127.0.98.3:35709/ using document root <none> and password file <none>
I20260620 14:30:07.598670 19366 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:07.598716 19366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:07.598865 19366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:07.600386 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "64cb620edded4949aabb44ee75098fac"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-hfc3"
server_key: "7381d3b27110d667b5b3308fe6ff0a7d"
server_key_iv: "df910a692e64b6852517a391c1c83035"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.600871 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "64cb620edded4949aabb44ee75098fac"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-hfc3"
server_key: "7381d3b27110d667b5b3308fe6ff0a7d"
server_key_iv: "df910a692e64b6852517a391c1c83035"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.602918 19366 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 14:30:07.603924 19381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.604090 19366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:07.604233 19366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "64cb620edded4949aabb44ee75098fac"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-hfc3"
server_key: "7381d3b27110d667b5b3308fe6ff0a7d"
server_key_iv: "df910a692e64b6852517a391c1c83035"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.604368 19366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:07.621860 19366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:07.623958 19366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:07.624125 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:07.624382 19366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:07.624696 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:07.624763 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.624809 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:07.624838 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.632207 19366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35271
I20260620 14:30:07.632311 19494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35271 every 8 connection(s)
I20260620 14:30:07.632664 19366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:07.636332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 19366
I20260620 14:30:07.636448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:07.636636 392 external_mini_cluster.cc:1428] Setting key 59abf9985b3afc4d9f991aa5ccd52057
Jun 20 14:30:07 dist-test-slave-hfc3 krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:07.643132 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.633520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44969 (local address 127.0.98.62:41565)
0620 14:30:07.633583 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.633586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.633833 (+ 247us) server_negotiation.cc:407] Connection header received
0620 14:30:07.635036 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.635037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.635049 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.635112 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.635806 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.636332 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.637850 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.638039 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.638097 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.640386 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.640388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.640388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.640395 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.641631 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.642037 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.642038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.642038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.642104 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.642432 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.642438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.642438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.642567 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.642636 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.642971 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.643050 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:30:07.643225 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.633515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41565 (local address 127.0.98.3:44969)
0620 14:30:07.633683 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.633734 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.634914 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.635170 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.635173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.635488 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:07.635696 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.635701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.636439 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.636441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.637780 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.637782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.638113 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.638114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.638218 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.638770 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:07.638779 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:07.640295 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:07.641705 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.641711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.641713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.641941 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.642180 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.642181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.642181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.642297 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.642637 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:07.642642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:07.642832 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.642917 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.643047 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260620 14:30:07.643766 19495 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41565
I20260620 14:30:07.643926 19495 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:07.644163 19495 heartbeater.cc:511] Master 127.0.98.62:41565 requested a full tablet report, sending...
I20260620 14:30:07.644584 19036 ts_manager.cc:194] Registered new tserver with Master: 64cb620edded4949aabb44ee75098fac (127.0.98.3:35271)
I20260620 14:30:07.645013 19036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44969
I20260620 14:30:07.648422 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:30:07.656194 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.649759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.0.98.62:41565)
0620 14:30:07.649825 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.649827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.649991 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:30:07.650151 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.650153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.650172 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.650237 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.650881 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.651368 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.652060 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.652241 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.652299 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.653435 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.653436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.653437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.653443 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.655049 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.655372 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.655373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.655373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.655426 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.655624 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.655624 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.655625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.655787 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.655862 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.655967 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.656027 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:30:07.658277 19036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:07.659904 19036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:07.667584 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.662761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35271 (local address 127.0.0.1:45722)
0620 14:30:07.662908 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.662953 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.663031 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.663501 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.663504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.663507 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:07.663588 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.663590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.664939 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.664940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.666289 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.666292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.667226 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.667227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.667326 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.667343 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.667432 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.667498 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260620 14:30:07.667836 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.662761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41195 (local address 127.0.0.1:39168)
0620 14:30:07.663485 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.663497 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.663547 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.663878 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.663879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.663881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:07.664038 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.664041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.665380 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.665381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.666608 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.666611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.667610 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.667611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.667671 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.667675 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.667730 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.667765 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":33,"thread_start_us":78,"threads_started":1}
I20260620 14:30:07.668005 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.663221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.0.98.2:41657)
0620 14:30:07.663446 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.663451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.663691 (+ 240us) server_negotiation.cc:407] Connection header received
0620 14:30:07.663820 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.663822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.663838 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.663889 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:07.664135 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.665214 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.666841 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.667543 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.667587 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.667643 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.667874 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.667921 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
I20260620 14:30:07.668017 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.662948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.0.98.3:35271)
0620 14:30:07.663199 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.663205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.663216 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:30:07.663269 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.663272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.663290 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.663375 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:07.663664 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.664844 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.666414 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.667139 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.667206 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.667291 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.667772 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.667826 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
I20260620 14:30:07.668038 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.663446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.0.98.1:41195)
0620 14:30:07.663642 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.663647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.663658 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:30:07.663725 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.663727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.663744 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.663960 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:07.664152 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.665267 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.666711 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.667463 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.667522 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.667576 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.667903 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.667950 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260620 14:30:07.668285 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.662982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41657 (local address 127.0.0.1:36564)
0620 14:30:07.663626 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.663636 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.663720 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.663996 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.663997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.663999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:07.664041 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.664043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.665340 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.665341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.666741 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.666744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.667702 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.667703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.667780 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.667784 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.668167 (+ 383us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.668212 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":41,"thread_start_us":59,"threads_started":1}
I20260620 14:30:07.668803 19429 tablet_service.cc:1467] Processing CreateTablet for tablet e06b582037b240a58a8a5266f28daf78 (DEFAULT_TABLE table=test-table [id=5ca376a0629145d1bc3076f92afc343c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:07.669054 19161 tablet_service.cc:1467] Processing CreateTablet for tablet e06b582037b240a58a8a5266f28daf78 (DEFAULT_TABLE table=test-table [id=5ca376a0629145d1bc3076f92afc343c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:07.669209 19429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06b582037b240a58a8a5266f28daf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:07.669179 19295 tablet_service.cc:1467] Processing CreateTablet for tablet e06b582037b240a58a8a5266f28daf78 (DEFAULT_TABLE table=test-table [id=5ca376a0629145d1bc3076f92afc343c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:07.669402 19161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06b582037b240a58a8a5266f28daf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:07.669443 19295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06b582037b240a58a8a5266f28daf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:07.671859 19512 tablet_bootstrap.cc:492] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac: Bootstrap starting.
I20260620 14:30:07.672514 19512 tablet_bootstrap.cc:654] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:07.672935 19513 tablet_bootstrap.cc:492] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9: Bootstrap starting.
I20260620 14:30:07.673098 19512 log.cc:824] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:07.673148 19514 tablet_bootstrap.cc:492] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f: Bootstrap starting.
I20260620 14:30:07.673736 19513 tablet_bootstrap.cc:654] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:07.673925 19514 tablet_bootstrap.cc:654] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:07.673955 19512 tablet_bootstrap.cc:492] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac: No bootstrap required, opened a new log
I20260620 14:30:07.674021 19512 ts_tablet_manager.cc:1397] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:07.674283 19513 log.cc:824] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:07.674664 19514 log.cc:824] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:07.675082 19513 tablet_bootstrap.cc:492] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9: No bootstrap required, opened a new log
I20260620 14:30:07.675199 19513 ts_tablet_manager.cc:1397] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:07.675482 19514 tablet_bootstrap.cc:492] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f: No bootstrap required, opened a new log
I20260620 14:30:07.675555 19514 ts_tablet_manager.cc:1397] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:07.676169 19512 raft_consensus.cc:348] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cb620edded4949aabb44ee75098fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "fafac1af20614c98ac908ab34ddcdd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41195 } } peers { permanent_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } }
I20260620 14:30:07.676373 19512 raft_consensus.cc:374] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:07.676414 19512 raft_consensus.cc:729] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64cb620edded4949aabb44ee75098fac, State: Initialized, Role: FOLLOWER
I20260620 14:30:07.676556 19512 consensus_queue.cc:260] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cb620edded4949aabb44ee75098fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "fafac1af20614c98ac908ab34ddcdd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41195 } } peers { permanent_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } }
I20260620 14:30:07.676832 19512 ts_tablet_manager.cc:1428] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:07.676810 19513 raft_consensus.cc:348] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cb620edded4949aabb44ee75098fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "fafac1af20614c98ac908ab34ddcdd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41195 } } peers { permanent_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } }
I20260620 14:30:07.676959 19495 heartbeater.cc:503] Master 127.0.98.62:41565 was elected leader, sending a full tablet report...
I20260620 14:30:07.676980 19513 raft_consensus.cc:374] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:07.677024 19513 raft_consensus.cc:729] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf60f1dcc18a43dda0ff0aba25635ab9, State: Initialized, Role: FOLLOWER
I20260620 14:30:07.677145 19513 consensus_queue.cc:260] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cb620edded4949aabb44ee75098fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "fafac1af20614c98ac908ab34ddcdd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41195 } } peers { permanent_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } }
I20260620 14:30:07.677414 19513 ts_tablet_manager.cc:1428] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:07.677451 19361 heartbeater.cc:503] Master 127.0.98.62:41565 was elected leader, sending a full tablet report...
I20260620 14:30:07.677659 19514 raft_consensus.cc:348] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cb620edded4949aabb44ee75098fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "fafac1af20614c98ac908ab34ddcdd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41195 } } peers { permanent_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } }
I20260620 14:30:07.677877 19514 raft_consensus.cc:374] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:07.677929 19514 raft_consensus.cc:729] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fafac1af20614c98ac908ab34ddcdd8f, State: Initialized, Role: FOLLOWER
I20260620 14:30:07.678053 19514 consensus_queue.cc:260] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cb620edded4949aabb44ee75098fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "fafac1af20614c98ac908ab34ddcdd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41195 } } peers { permanent_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } }
I20260620 14:30:07.678380 19514 ts_tablet_manager.cc:1428] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:07.678436 19227 heartbeater.cc:503] Master 127.0.98.62:41565 was elected leader, sending a full tablet report...
W20260620 14:30:07.716127 19362 tablet.cc:2369] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:30:07.805931 19228 tablet.cc:2369] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:30:07.861516 19519 raft_consensus.cc:482] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:07.861642 19519 raft_consensus.cc:504] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cb620edded4949aabb44ee75098fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "fafac1af20614c98ac908ab34ddcdd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41195 } } peers { permanent_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } }
I20260620 14:30:07.862017 19519 leader_election.cc:288] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64cb620edded4949aabb44ee75098fac (127.0.98.3:35271), fafac1af20614c98ac908ab34ddcdd8f (127.0.98.1:41195)
I20260620 14:30:07.865963 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.862387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53437 (local address 127.0.98.3:35271)
0620 14:30:07.862458 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.862460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.862468 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:30:07.862561 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.862562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.862568 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.862608 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:07.862739 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.863462 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.865036 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.865612 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.865675 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.865683 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.865856 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.865893 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:30:07.865963 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.862256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55495 (local address 127.0.98.1:41195)
0620 14:30:07.862304 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.862307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.862355 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:30:07.862388 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.862389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.862401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.862432 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:07.862683 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.863462 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.865036 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.865588 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.865672 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.865684 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.865856 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.865893 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:30:07.865975 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.862181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41195 (local address 127.0.98.2:55495)
0620 14:30:07.862221 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.862233 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.862285 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.862532 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.862533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.862535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:07.862606 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.862609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.863563 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.863564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.864939 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.864943 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.865688 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.865689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.865776 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.865779 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.865838 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.865881 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":118}
I20260620 14:30:07.865980 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.862200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35271 (local address 127.0.98.2:53437)
0620 14:30:07.862421 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.862429 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.862470 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.862619 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.862620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.862621 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:07.862676 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.862678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.863577 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.863577 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.864939 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.864943 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.865688 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.865689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.865774 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.865779 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.865838 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.865881 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":57,"thread_start_us":131,"threads_started":1}
I20260620 14:30:07.866441 19181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06b582037b240a58a8a5266f28daf78" candidate_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fafac1af20614c98ac908ab34ddcdd8f" is_pre_election: true
I20260620 14:30:07.866611 19181 raft_consensus.cc:2454] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf60f1dcc18a43dda0ff0aba25635ab9 in term 0.
I20260620 14:30:07.866860 19249 leader_election.cc:302] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf60f1dcc18a43dda0ff0aba25635ab9, fafac1af20614c98ac908ab34ddcdd8f; no voters:
I20260620 14:30:07.867008 19519 raft_consensus.cc:2781] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:07.867066 19519 raft_consensus.cc:482] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:07.867091 19519 raft_consensus.cc:3037] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:07.867038 19442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06b582037b240a58a8a5266f28daf78" candidate_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64cb620edded4949aabb44ee75098fac" is_pre_election: true
I20260620 14:30:07.867175 19442 raft_consensus.cc:2454] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf60f1dcc18a43dda0ff0aba25635ab9 in term 0.
I20260620 14:30:07.867815 19519 raft_consensus.cc:504] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cb620edded4949aabb44ee75098fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "fafac1af20614c98ac908ab34ddcdd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41195 } } peers { permanent_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } }
I20260620 14:30:07.867961 19519 leader_election.cc:288] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [CANDIDATE]: Term 1 election: Requested vote from peers 64cb620edded4949aabb44ee75098fac (127.0.98.3:35271), fafac1af20614c98ac908ab34ddcdd8f (127.0.98.1:41195)
I20260620 14:30:07.868098 19442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06b582037b240a58a8a5266f28daf78" candidate_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64cb620edded4949aabb44ee75098fac"
I20260620 14:30:07.868149 19181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06b582037b240a58a8a5266f28daf78" candidate_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fafac1af20614c98ac908ab34ddcdd8f"
I20260620 14:30:07.868178 19442 raft_consensus.cc:3037] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:07.868227 19181 raft_consensus.cc:3037] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:07.869071 19181 raft_consensus.cc:2454] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf60f1dcc18a43dda0ff0aba25635ab9 in term 1.
I20260620 14:30:07.869071 19442 raft_consensus.cc:2454] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf60f1dcc18a43dda0ff0aba25635ab9 in term 1.
I20260620 14:30:07.869283 19248 leader_election.cc:302] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64cb620edded4949aabb44ee75098fac, bf60f1dcc18a43dda0ff0aba25635ab9; no voters:
I20260620 14:30:07.869403 19519 raft_consensus.cc:2781] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:07.869575 19519 raft_consensus.cc:686] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 1 LEADER]: Becoming Leader. State: Replica: bf60f1dcc18a43dda0ff0aba25635ab9, State: Running, Role: LEADER
I20260620 14:30:07.869681 19519 consensus_queue.cc:237] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cb620edded4949aabb44ee75098fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } } peers { permanent_uuid: "fafac1af20614c98ac908ab34ddcdd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41195 } } peers { permanent_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } }
I20260620 14:30:07.870569 19035 catalog_manager.cc:5495] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf60f1dcc18a43dda0ff0aba25635ab9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cb620edded4949aabb44ee75098fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fafac1af20614c98ac908ab34ddcdd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf60f1dcc18a43dda0ff0aba25635ab9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } health_report { overall_health: HEALTHY } } }
W20260620 14:30:07.883219 19496 tablet.cc:2369] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:30:07.892475 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.890163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.0.98.2:41657)
0620 14:30:07.890224 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.890226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.890234 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:30:07.890317 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.890318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.890407 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.890469 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:07.890598 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.891059 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.891786 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.891964 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.892013 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.892145 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:07.892243 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:07.892302 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.892336 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.892352 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:30:07.894485 19442 raft_consensus.cc:1261] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac [term 1 FOLLOWER]: Refusing update from remote peer bf60f1dcc18a43dda0ff0aba25635ab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:07.894524 19181 raft_consensus.cc:1261] T e06b582037b240a58a8a5266f28daf78 P fafac1af20614c98ac908ab34ddcdd8f [term 1 FOLLOWER]: Refusing update from remote peer bf60f1dcc18a43dda0ff0aba25635ab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:07.894865 19522 consensus_queue.cc:1035] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64cb620edded4949aabb44ee75098fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:07.894999 19519 consensus_queue.cc:1035] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fafac1af20614c98ac908ab34ddcdd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:07.897413 19525 mvcc.cc:205] Tried to move back new op lower bound from 7298931949132517376 to 7298931949034921984. Current Snapshot: MvccSnapshot[applied={T|T < 7298931949132517376}]
I20260620 14:30:07.900362 19036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:07.900444 19036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:07.901165 19036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92558fc022dd49c982b6f1dffb703a4e: Sending DeleteTablet for 3 replicas of tablet e06b582037b240a58a8a5266f28daf78
I20260620 14:30:07.901484 19429 tablet_service.cc:1514] Processing DeleteTablet for tablet e06b582037b240a58a8a5266f28daf78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45722
I20260620 14:30:07.901583 19161 tablet_service.cc:1514] Processing DeleteTablet for tablet e06b582037b240a58a8a5266f28daf78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39168
I20260620 14:30:07.901775 19534 tablet_replica.cc:330] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac: stopping tablet replica
I20260620 14:30:07.901785 19295 tablet_service.cc:1514] Processing DeleteTablet for tablet e06b582037b240a58a8a5266f28daf78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36564
I20260620 14:30:07.901911 19534 raft_consensus.cc:2229] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:07.902017 19534 raft_consensus.cc:2258] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:07.902032 19536 tablet_replica.cc:330] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9: stopping tablet replica
I20260620 14:30:07.902130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 19098
I20260620 14:30:07.902184 19536 raft_consensus.cc:2229] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:30:07.902307 19536 raft_consensus.cc:2258] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:07.903139 19534 ts_tablet_manager.cc:1905] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:07.903800 19536 ts_tablet_manager.cc:1905] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:07.905026 19534 ts_tablet_manager.cc:1918] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:07.905164 19534 log.cc:1196] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/e06b582037b240a58a8a5266f28daf78
I20260620 14:30:07.905505 19534 ts_tablet_manager.cc:1939] T e06b582037b240a58a8a5266f28daf78 P 64cb620edded4949aabb44ee75098fac: Deleting consensus metadata
I20260620 14:30:07.905815 19536 ts_tablet_manager.cc:1918] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:07.905892 19536 log.cc:1196] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/e06b582037b240a58a8a5266f28daf78
I20260620 14:30:07.906116 19020 catalog_manager.cc:4841] TS 64cb620edded4949aabb44ee75098fac (127.0.98.3:35271): tablet e06b582037b240a58a8a5266f28daf78 (table test-table [id=5ca376a0629145d1bc3076f92afc343c]) successfully deleted
I20260620 14:30:07.906311 19536 ts_tablet_manager.cc:1939] T e06b582037b240a58a8a5266f28daf78 P bf60f1dcc18a43dda0ff0aba25635ab9: Deleting consensus metadata
I20260620 14:30:07.906769 19021 catalog_manager.cc:4841] TS bf60f1dcc18a43dda0ff0aba25635ab9 (127.0.98.2:41657): tablet e06b582037b240a58a8a5266f28daf78 (table test-table [id=5ca376a0629145d1bc3076f92afc343c]) successfully deleted
W20260620 14:30:07.909972 19021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41195 (error 108)
I20260620 14:30:07.910099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 19232
I20260620 14:30:07.911242 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.911092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41195 (local address 127.0.0.1:39182)
0620 14:30:07.911159 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.911200 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 14:30:07.911370 19021 catalog_manager.cc:4584] TS fafac1af20614c98ac908ab34ddcdd8f (127.0.98.1:41195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e06b582037b240a58a8a5266f28daf78: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41195: connect: Connection refused (error 111)
I20260620 14:30:07.914898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 19366
I20260620 14:30:07.919206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 19004
2026-06-20T14:30:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2140 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T14:30:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:07Z Disabled control of system clock
I20260620 14:30:07.947467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34845
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:30:08.028771 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:08.029007 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:08.029028 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:08.030117 19541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:08.030153 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:08.030165 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:08.030179 19541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:08.030190 19541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:08.031646 19541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:08.031857 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:08.032104 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:08.038030 19549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:08.038035 19547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:08.038056 19546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:08.038162 19541 server_base.cc:1034] running on GCE node
I20260620 14:30:08.038538 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:08.038895 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:08.040076 19541 hybrid_clock.cc:648] HybridClock initialized: now 1781965808040073 us; error 50 us; skew 500 ppm
I20260620 14:30:08.041291 19541 webserver.cc:460] Webserver started at http://127.0.98.62:33779/ using document root <none> and password file <none>
I20260620 14:30:08.041558 19541 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:08.041630 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:08.041800 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:08.043010 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "8406fb8ed7de43a6b9dd3746d96f1082"
format_stamp: "Formatted at 2026-06-20 14:30:08 on dist-test-slave-hfc3"
server_key: "0487f01255e74f36acde0549124708b2"
server_key_iv: "77cc41edb2a55e553f21d9a6b5476093"
server_key_version: "encryptionkey@0"
I20260620 14:30:08.043386 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "8406fb8ed7de43a6b9dd3746d96f1082"
format_stamp: "Formatted at 2026-06-20 14:30:08 on dist-test-slave-hfc3"
server_key: "0487f01255e74f36acde0549124708b2"
server_key_iv: "77cc41edb2a55e553f21d9a6b5476093"
server_key_version: "encryptionkey@0"
I20260620 14:30:08.044689 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:30:08.045408 19555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:08.045581 19541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:08.045680 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "8406fb8ed7de43a6b9dd3746d96f1082"
format_stamp: "Formatted at 2026-06-20 14:30:08 on dist-test-slave-hfc3"
server_key: "0487f01255e74f36acde0549124708b2"
server_key_iv: "77cc41edb2a55e553f21d9a6b5476093"
server_key_version: "encryptionkey@0"
I20260620 14:30:08.045770 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:08.063347 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:08.063745 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:08.063921 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:08.068025 19541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34845
I20260620 14:30:08.068074 19607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34845 every 8 connection(s)
I20260620 14:30:08.068470 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:08.069525 19608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:08.071842 19608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082: Bootstrap starting.
I20260620 14:30:08.072309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 19541
I20260620 14:30:08.072451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:08.072675 392 external_mini_cluster.cc:1428] Setting key 2eadda387fcd651c86f42f63386d2298
I20260620 14:30:08.072801 19608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:08.073177 19608 log.cc:824] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:08.074046 19608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082: No bootstrap required, opened a new log
I20260620 14:30:08.076422 19608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8406fb8ed7de43a6b9dd3746d96f1082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34845 } }
I20260620 14:30:08.076586 19608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:08.076633 19608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8406fb8ed7de43a6b9dd3746d96f1082, State: Initialized, Role: FOLLOWER
I20260620 14:30:08.076768 19608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8406fb8ed7de43a6b9dd3746d96f1082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34845 } }
I20260620 14:30:08.076849 19608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:08.076884 19608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:08.076922 19608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:08.077613 19608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8406fb8ed7de43a6b9dd3746d96f1082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34845 } }
I20260620 14:30:08.077732 19608 leader_election.cc:302] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8406fb8ed7de43a6b9dd3746d96f1082; no voters:
I20260620 14:30:08.077899 19608 leader_election.cc:288] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:08.077961 19613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:08.078125 19613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [term 1 LEADER]: Becoming Leader. State: Replica: 8406fb8ed7de43a6b9dd3746d96f1082, State: Running, Role: LEADER
I20260620 14:30:08.078151 19608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:08.078253 19613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8406fb8ed7de43a6b9dd3746d96f1082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34845 } }
I20260620 14:30:08.078661 19615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8406fb8ed7de43a6b9dd3746d96f1082. Latest consensus state: current_term: 1 leader_uuid: "8406fb8ed7de43a6b9dd3746d96f1082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8406fb8ed7de43a6b9dd3746d96f1082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34845 } } }
I20260620 14:30:08.078748 19615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:08.078967 19620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:08.079411 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.073268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.0.98.62:34845)
0620 14:30:08.073470 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.073479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.073518 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:30:08.074053 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.074061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.074140 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.074307 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.074592 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.076005 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.076826 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.077448 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.077584 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.078394 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:08.078398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:08.078399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:08.078614 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:08.078661 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:08.078679 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:08.078740 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:08.079073 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:30:08.079157 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260620 14:30:08.079480 19620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:08.080078 19614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8406fb8ed7de43a6b9dd3746d96f1082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8406fb8ed7de43a6b9dd3746d96f1082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34845 } } }
I20260620 14:30:08.080192 19614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:08.080873 19620 catalog_manager.cc:1347] Generated new cluster ID: 98fe1e4b74474d8d9cadc1e7e60b2e0d
I20260620 14:30:08.080924 19620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:08.093912 19620 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:08.094568 19620 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:08.099179 19620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082: Generated new TSK 0
I20260620 14:30:08.099383 19620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:08.101691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34845
--builtin_ntp_servers=127.0.98.20:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:30:08.201290 19632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:08.201543 19632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:08.201582 19632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:08.202845 19632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:08.202908 19632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:08.202952 19632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:30:08.204356 19632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:08.204645 19632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:08.204959 19632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:08.210027 19637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:08.210036 19640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:08.210037 19638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:08.210265 19632 server_base.cc:1034] running on GCE node
I20260620 14:30:08.210583 19632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:08.210875 19632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:08.212049 19632 hybrid_clock.cc:648] HybridClock initialized: now 1781965808212023 us; error 40 us; skew 500 ppm
I20260620 14:30:08.213434 19632 webserver.cc:460] Webserver started at http://127.0.98.1:38199/ using document root <none> and password file <none>
I20260620 14:30:08.213680 19632 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:08.213776 19632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:08.213922 19632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:08.215189 19632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/data/instance:
uuid: "13a36a5a60a94d9eb01f63ca1fc28de8"
format_stamp: "Formatted at 2026-06-20 14:30:08 on dist-test-slave-hfc3"
server_key: "47388b9a6e1f6b2e291cfce4fad716f3"
server_key_iv: "1444adc1fb2c86885582c1354aa2f649"
server_key_version: "encryptionkey@0"
I20260620 14:30:08.215662 19632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "13a36a5a60a94d9eb01f63ca1fc28de8"
format_stamp: "Formatted at 2026-06-20 14:30:08 on dist-test-slave-hfc3"
server_key: "47388b9a6e1f6b2e291cfce4fad716f3"
server_key_iv: "1444adc1fb2c86885582c1354aa2f649"
server_key_version: "encryptionkey@0"
I20260620 14:30:08.217046 19632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:30:08.217851 19646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:08.218022 19632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:08.218107 19632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/wal
uuid: "13a36a5a60a94d9eb01f63ca1fc28de8"
format_stamp: "Formatted at 2026-06-20 14:30:08 on dist-test-slave-hfc3"
server_key: "47388b9a6e1f6b2e291cfce4fad716f3"
server_key_iv: "1444adc1fb2c86885582c1354aa2f649"
server_key_version: "encryptionkey@0"
I20260620 14:30:08.218178 19632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:08.227672 19632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:08.227978 19632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:08.228130 19632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:08.228425 19632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:08.228826 19632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:08.228863 19632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:08.228886 19632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:08.228899 19632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:08.236348 19632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34145
I20260620 14:30:08.236387 19759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34145 every 8 connection(s)
I20260620 14:30:08.236748 19632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:08.237895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 19632
I20260620 14:30:08.238004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:08.238195 392 external_mini_cluster.cc:1428] Setting key 6d12a1b0443541040336d6ced0fd3cd9
I20260620 14:30:08.239193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34845
--builtin_ntp_servers=127.0.98.20:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:30:08.243494 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.237515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53459 (local address 127.0.98.62:34845)
0620 14:30:08.237568 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.237570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.237911 (+ 341us) server_negotiation.cc:407] Connection header received
0620 14:30:08.238639 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.238640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.238648 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.238756 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.239634 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.240155 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.241744 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.242059 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.242143 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.243153 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:08.243155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:08.243155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:08.243161 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:08.243179 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:08.243184 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:08.243250 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:08.243347 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:30:08.243375 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:30:08.243649 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.237497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34845 (local address 127.0.98.1:53459)
0620 14:30:08.237736 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:08.237786 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:08.238540 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:08.238831 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:08.238836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:08.238847 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:08.239521 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.239527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.240260 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.240261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.241637 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.241640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.242152 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.242153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.242286 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.242958 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:08.242959 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:08.242990 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:08.243016 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:08.243277 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:08.243282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:08.243284 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:30:08.243343 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:30:08.243430 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260620 14:30:08.244153 19760 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34845
I20260620 14:30:08.244270 19760 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:08.244499 19760 heartbeater.cc:511] Master 127.0.98.62:34845 requested a full tablet report, sending...
I20260620 14:30:08.245070 19572 ts_manager.cc:194] Registered new tserver with Master: 13a36a5a60a94d9eb01f63ca1fc28de8 (127.0.98.1:34145)
I20260620 14:30:08.245748 19572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53459
W20260620 14:30:08.321801 19763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:08.322024 19763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:08.322048 19763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:08.323446 19763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:08.323499 19763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:08.323526 19763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:30:08.324951 19763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:08.325220 19763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:08.325536 19763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:08.330612 19771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:08.330590 19769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:08.330591 19768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:08.330998 19763 server_base.cc:1034] running on GCE node
I20260620 14:30:08.331282 19763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:08.331552 19763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:08.332760 19763 hybrid_clock.cc:648] HybridClock initialized: now 1781965808332729 us; error 46 us; skew 500 ppm
I20260620 14:30:08.334084 19763 webserver.cc:460] Webserver started at http://127.0.98.2:33519/ using document root <none> and password file <none>
I20260620 14:30:08.334453 19763 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:08.334528 19763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:08.334681 19763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:08.335804 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/data/instance:
uuid: "bc3705e68d444736998efff7987a9f5f"
format_stamp: "Formatted at 2026-06-20 14:30:08 on dist-test-slave-hfc3"
server_key: "46aaed293cf3a6f5dcba3e6d47a270af"
server_key_iv: "52db3f0b2ab75a409ded88706d769787"
server_key_version: "encryptionkey@0"
I20260620 14:30:08.336202 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bc3705e68d444736998efff7987a9f5f"
format_stamp: "Formatted at 2026-06-20 14:30:08 on dist-test-slave-hfc3"
server_key: "46aaed293cf3a6f5dcba3e6d47a270af"
server_key_iv: "52db3f0b2ab75a409ded88706d769787"
server_key_version: "encryptionkey@0"
I20260620 14:30:08.337656 19763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:08.338712 19777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:08.338928 19763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:30:08.339039 19763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/wal
uuid: "bc3705e68d444736998efff7987a9f5f"
format_stamp: "Formatted at 2026-06-20 14:30:08 on dist-test-slave-hfc3"
server_key: "46aaed293cf3a6f5dcba3e6d47a270af"
server_key_iv: "52db3f0b2ab75a409ded88706d769787"
server_key_version: "encryptionkey@0"
I20260620 14:30:08.339120 19763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:08.360311 19763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:08.360651 19763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:08.360812 19763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:08.361065 19763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:08.361433 19763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:08.361467 19763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:08.361491 19763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:08.361538 19763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:08.367434 19763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36423
I20260620 14:30:08.367520 19890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36423 every 8 connection(s)
I20260620 14:30:08.367861 19763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:08.372957 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.368629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47575 (local address 127.0.98.62:34845)
0620 14:30:08.368739 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.368741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.368942 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:30:08.369430 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.369432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.369442 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.369498 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.370178 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.370736 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.371669 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.371860 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.371914 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.372756 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:08.372757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:08.372757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:08.372762 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:08.372772 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:08.372775 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:08.372813 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:08.372881 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:30:08.372900 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:30:08.373170 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.368591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34845 (local address 127.0.98.2:47575)
0620 14:30:08.368813 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:08.368849 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:08.369342 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:08.369576 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:08.369579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:08.369587 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:08.370102 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.370106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.370815 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.370816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.371590 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.371592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.371932 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.371933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.372040 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.372618 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:08.372618 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:08.372649 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:08.372672 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:08.372816 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:08.372818 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:08.372821 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:08.372869 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:30:08.372947 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":115,"threads_started":1}
I20260620 14:30:08.373715 19891 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34845
I20260620 14:30:08.373826 19891 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:08.374004 19891 heartbeater.cc:511] Master 127.0.98.62:34845 requested a full tablet report, sending...
I20260620 14:30:08.374444 19572 ts_manager.cc:194] Registered new tserver with Master: bc3705e68d444736998efff7987a9f5f (127.0.98.2:36423)
I20260620 14:30:08.374832 19572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47575
I20260620 14:30:08.375207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 19763
I20260620 14:30:08.375317 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:08.375510 392 external_mini_cluster.cc:1428] Setting key 6c80c70316d98cdff69014476d885a85
I20260620 14:30:08.376360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34845
--builtin_ntp_servers=127.0.98.20:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:30:08.457800 19894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:08.458029 19894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:08.458050 19894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:08.459270 19894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:08.459317 19894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:08.459343 19894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:30:08.460760 19894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:08.461017 19894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:08.461356 19894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:08.466447 19902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:08.466424 19900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:08.466441 19899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:08.466729 19894 server_base.cc:1034] running on GCE node
I20260620 14:30:08.466957 19894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:08.467170 19894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:08.468340 19894 hybrid_clock.cc:648] HybridClock initialized: now 1781965808468327 us; error 31 us; skew 500 ppm
I20260620 14:30:08.469658 19894 webserver.cc:460] Webserver started at http://127.0.98.3:43153/ using document root <none> and password file <none>
I20260620 14:30:08.469916 19894 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:08.469988 19894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:08.470151 19894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:08.471329 19894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/data/instance:
uuid: "bd745259718940db8d10b3a205dd8470"
format_stamp: "Formatted at 2026-06-20 14:30:08 on dist-test-slave-hfc3"
server_key: "cf4217d7e2744762597af2063a90c9ac"
server_key_iv: "81cb199711b51d8f92bf30ccf1d1e3ee"
server_key_version: "encryptionkey@0"
I20260620 14:30:08.471701 19894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bd745259718940db8d10b3a205dd8470"
format_stamp: "Formatted at 2026-06-20 14:30:08 on dist-test-slave-hfc3"
server_key: "cf4217d7e2744762597af2063a90c9ac"
server_key_iv: "81cb199711b51d8f92bf30ccf1d1e3ee"
server_key_version: "encryptionkey@0"
I20260620 14:30:08.473028 19894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:30:08.473834 19908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:08.474031 19894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:08.474130 19894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/wal
uuid: "bd745259718940db8d10b3a205dd8470"
format_stamp: "Formatted at 2026-06-20 14:30:08 on dist-test-slave-hfc3"
server_key: "cf4217d7e2744762597af2063a90c9ac"
server_key_iv: "81cb199711b51d8f92bf30ccf1d1e3ee"
server_key_version: "encryptionkey@0"
I20260620 14:30:08.474246 19894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:08.495007 19894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:08.495340 19894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:08.495496 19894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:08.495743 19894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:08.496088 19894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:08.496146 19894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:08.496182 19894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:08.496209 19894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:08.502035 19894 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38209
I20260620 14:30:08.502132 20021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38209 every 8 connection(s)
I20260620 14:30:08.502513 19894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:08.507519 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.503309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36211 (local address 127.0.98.62:34845)
0620 14:30:08.503363 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.503365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.503521 (+ 156us) server_negotiation.cc:407] Connection header received
0620 14:30:08.504036 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.504037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.504047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.504131 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.504787 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.505287 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.506252 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.506462 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.506525 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.507301 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:08.507302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:08.507303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:08.507307 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:08.507321 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:08.507325 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:08.507375 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:08.507437 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:30:08.507459 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:30:08.507702 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.503209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34845 (local address 127.0.98.3:36211)
0620 14:30:08.503400 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:08.503443 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:08.503941 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:08.504187 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:08.504190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:08.504198 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:08.504685 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.504689 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.505368 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.505369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.506142 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.506144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.506541 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.506542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.506651 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.507166 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:08.507167 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:08.507194 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:08.507216 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:08.507378 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:08.507380 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:08.507382 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:30:08.507427 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:30:08.507503 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260620 14:30:08.508174 20022 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34845
I20260620 14:30:08.508261 20022 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:08.508450 20022 heartbeater.cc:511] Master 127.0.98.62:34845 requested a full tablet report, sending...
I20260620 14:30:08.508823 19572 ts_manager.cc:194] Registered new tserver with Master: bd745259718940db8d10b3a205dd8470 (127.0.98.3:38209)
I20260620 14:30:08.509274 19572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36211
I20260620 14:30:08.511762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 19894
I20260620 14:30:08.511845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:08.511998 392 external_mini_cluster.cc:1428] Setting key e5683dfdc85e6d487350d82c10bae386
I20260620 14:30:08.512997 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:30:08.517242 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.514486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.0.98.62:34845)
0620 14:30:08.514542 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.514544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.514675 (+ 131us) server_negotiation.cc:407] Connection header received
0620 14:30:08.514716 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.514717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.514729 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.514786 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.514917 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.515409 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.516225 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.516410 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.516476 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.517046 (+ 570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:08.517047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:08.517047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:08.517052 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:08.517066 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:08.517070 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:08.517117 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:08.517168 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:30:08.517187 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:30:08.518878 19572 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:08.520462 19572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:08.527340 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.522721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34145 (local address 127.0.0.1:50568)
0620 14:30:08.522855 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:08.522898 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:08.522962 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:08.523492 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:08.523495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:08.523498 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:08.523630 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.523632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.525040 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.525041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.526067 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.526070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.527030 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.527031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.527118 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.527144 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:30:08.527192 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:30:08.527247 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":21,"thread_start_us":79,"threads_started":1}
I20260620 14:30:08.527514 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.522784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36423 (local address 127.0.0.1:43098)
0620 14:30:08.523392 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:08.523404 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:08.523699 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:08.523994 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:08.523994 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:08.523996 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:08.524034 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.524036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.525363 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.525364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.526475 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.526477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.527268 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.527270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.527329 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.527333 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:30:08.527386 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:30:08.527429 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260620 14:30:08.527522 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.522910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.0.98.2:36423)
0620 14:30:08.523332 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.523336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.523452 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:30:08.523815 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.523818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.523835 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.523922 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:08.524124 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.525278 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.526558 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.527178 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.527269 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.527354 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:08.527410 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:30:08.527444 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":109,"threads_started":1}
I20260620 14:30:08.527629 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.522955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38209 (local address 127.0.0.1:47966)
0620 14:30:08.523572 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:08.523582 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:08.523644 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:08.523847 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:08.523847 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:08.523849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:08.523888 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.523890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.524941 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.524942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.526202 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.526204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.527248 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.527248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.527308 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.527311 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:08.527358 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:30:08.527398 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":70,"threads_started":1}
I20260620 14:30:08.528031 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.523123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.0.98.1:34145)
0620 14:30:08.523289 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.523294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.523307 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:30:08.523352 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.523354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.523372 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.523469 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:08.523754 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.524933 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.526161 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.526925 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.527778 (+ 853us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.527878 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:08.527905 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 14:30:08.527949 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260620 14:30:08.528468 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.523057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.0.98.3:38209)
0620 14:30:08.523259 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.523264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.523630 (+ 366us) server_negotiation.cc:407] Connection header received
0620 14:30:08.523754 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.523756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.523773 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.523852 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:08.523945 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.524744 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.526347 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.527051 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.527115 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.527215 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:08.527958 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 14:30:08.528003 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":122,"threads_started":1}
I20260620 14:30:08.528537 19825 tablet_service.cc:1467] Processing CreateTablet for tablet 907f0a81e94246d28cb4e91676cbe09b (DEFAULT_TABLE table=test-table [id=0b379243902c4b2a9e7a526dbf5408ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:08.528944 19825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907f0a81e94246d28cb4e91676cbe09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:08.528846 19694 tablet_service.cc:1467] Processing CreateTablet for tablet 907f0a81e94246d28cb4e91676cbe09b (DEFAULT_TABLE table=test-table [id=0b379243902c4b2a9e7a526dbf5408ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:08.529168 19694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907f0a81e94246d28cb4e91676cbe09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:08.530931 19956 tablet_service.cc:1467] Processing CreateTablet for tablet 907f0a81e94246d28cb4e91676cbe09b (DEFAULT_TABLE table=test-table [id=0b379243902c4b2a9e7a526dbf5408ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:08.531291 19956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907f0a81e94246d28cb4e91676cbe09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:08.532505 20037 tablet_bootstrap.cc:492] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8: Bootstrap starting.
I20260620 14:30:08.533286 20037 tablet_bootstrap.cc:654] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:08.533610 20037 log.cc:824] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:08.533998 20036 tablet_bootstrap.cc:492] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f: Bootstrap starting.
I20260620 14:30:08.534394 20039 tablet_bootstrap.cc:492] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470: Bootstrap starting.
I20260620 14:30:08.534566 20037 tablet_bootstrap.cc:492] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8: No bootstrap required, opened a new log
I20260620 14:30:08.534641 20037 ts_tablet_manager.cc:1397] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:08.534824 20036 tablet_bootstrap.cc:654] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:08.535089 20039 tablet_bootstrap.cc:654] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:08.535118 20036 log.cc:824] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:08.535454 20039 log.cc:824] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:08.535904 20036 tablet_bootstrap.cc:492] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f: No bootstrap required, opened a new log
I20260620 14:30:08.535971 20036 ts_tablet_manager.cc:1397] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:30:08.536613 20039 tablet_bootstrap.cc:492] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470: No bootstrap required, opened a new log
I20260620 14:30:08.536686 20039 ts_tablet_manager.cc:1397] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:08.537420 20037 raft_consensus.cc:348] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34145 } } peers { permanent_uuid: "bd745259718940db8d10b3a205dd8470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38209 } } peers { permanent_uuid: "bc3705e68d444736998efff7987a9f5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36423 } }
I20260620 14:30:08.537629 20037 raft_consensus.cc:374] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:08.537686 20037 raft_consensus.cc:729] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a36a5a60a94d9eb01f63ca1fc28de8, State: Initialized, Role: FOLLOWER
I20260620 14:30:08.537802 20037 consensus_queue.cc:260] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34145 } } peers { permanent_uuid: "bd745259718940db8d10b3a205dd8470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38209 } } peers { permanent_uuid: "bc3705e68d444736998efff7987a9f5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36423 } }
I20260620 14:30:08.537878 20036 raft_consensus.cc:348] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34145 } } peers { permanent_uuid: "bd745259718940db8d10b3a205dd8470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38209 } } peers { permanent_uuid: "bc3705e68d444736998efff7987a9f5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36423 } }
I20260620 14:30:08.538061 20037 ts_tablet_manager.cc:1428] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:30:08.538064 20036 raft_consensus.cc:374] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:08.538110 20036 raft_consensus.cc:729] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3705e68d444736998efff7987a9f5f, State: Initialized, Role: FOLLOWER
I20260620 14:30:08.538268 19760 heartbeater.cc:503] Master 127.0.98.62:34845 was elected leader, sending a full tablet report...
I20260620 14:30:08.538230 20036 consensus_queue.cc:260] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34145 } } peers { permanent_uuid: "bd745259718940db8d10b3a205dd8470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38209 } } peers { permanent_uuid: "bc3705e68d444736998efff7987a9f5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36423 } }
I20260620 14:30:08.538494 20036 ts_tablet_manager.cc:1428] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:30:08.538610 19891 heartbeater.cc:503] Master 127.0.98.62:34845 was elected leader, sending a full tablet report...
I20260620 14:30:08.539378 20039 raft_consensus.cc:348] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34145 } } peers { permanent_uuid: "bd745259718940db8d10b3a205dd8470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38209 } } peers { permanent_uuid: "bc3705e68d444736998efff7987a9f5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36423 } }
I20260620 14:30:08.539515 20039 raft_consensus.cc:374] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:08.539557 20039 raft_consensus.cc:729] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd745259718940db8d10b3a205dd8470, State: Initialized, Role: FOLLOWER
I20260620 14:30:08.539662 20039 consensus_queue.cc:260] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34145 } } peers { permanent_uuid: "bd745259718940db8d10b3a205dd8470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38209 } } peers { permanent_uuid: "bc3705e68d444736998efff7987a9f5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36423 } }
I20260620 14:30:08.539891 20039 ts_tablet_manager.cc:1428] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:08.539968 20022 heartbeater.cc:503] Master 127.0.98.62:34845 was elected leader, sending a full tablet report...
I20260620 14:30:08.559128 20042 raft_consensus.cc:482] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:08.559257 20042 raft_consensus.cc:504] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34145 } } peers { permanent_uuid: "bd745259718940db8d10b3a205dd8470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38209 } } peers { permanent_uuid: "bc3705e68d444736998efff7987a9f5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36423 } }
I20260620 14:30:08.559685 20042 leader_election.cc:288] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd745259718940db8d10b3a205dd8470 (127.0.98.3:38209), bc3705e68d444736998efff7987a9f5f (127.0.98.2:36423)
I20260620 14:30:08.562489 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.559906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39773 (local address 127.0.98.3:38209)
0620 14:30:08.560020 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.560022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.560032 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:30:08.560067 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.560068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.560081 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.560145 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:08.560319 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.560816 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.561738 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.562094 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.562191 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.562201 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:08.562367 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:30:08.562413 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 14:30:08.562489 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.559843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38209 (local address 127.0.98.1:39773)
0620 14:30:08.559900 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:08.559913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:08.559957 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:08.560159 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:08.560160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:08.560162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:08.560240 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.560243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.560903 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.560904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.561644 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.561646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.562194 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.562195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.562267 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.562270 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:08.562384 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 14:30:08.562423 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:30:08.562637 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.559916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50305 (local address 127.0.98.2:36423)
0620 14:30:08.559966 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.559968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.560120 (+ 152us) server_negotiation.cc:407] Connection header received
0620 14:30:08.560260 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.560261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.560277 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.560328 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:08.560513 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.560972 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.561897 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.562261 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.562342 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.562373 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:08.562537 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:30:08.562575 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:30:08.562649 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.559853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36423 (local address 127.0.98.1:50305)
0620 14:30:08.560057 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:08.560069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:08.560147 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:08.560357 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:08.560358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:08.560359 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:08.560426 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.560428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.561056 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.561057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.561798 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:08.561800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.562370 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.562371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:08.562451 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.562454 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:08.562531 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:30:08.562568 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":128,"threads_started":1}
I20260620 14:30:08.562862 19976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907f0a81e94246d28cb4e91676cbe09b" candidate_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd745259718940db8d10b3a205dd8470" is_pre_election: true
I20260620 14:30:08.562950 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907f0a81e94246d28cb4e91676cbe09b" candidate_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3705e68d444736998efff7987a9f5f" is_pre_election: true
I20260620 14:30:08.563038 19976 raft_consensus.cc:2454] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13a36a5a60a94d9eb01f63ca1fc28de8 in term 0.
I20260620 14:30:08.563098 19845 raft_consensus.cc:2454] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13a36a5a60a94d9eb01f63ca1fc28de8 in term 0.
I20260620 14:30:08.563277 19649 leader_election.cc:302] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a36a5a60a94d9eb01f63ca1fc28de8, bd745259718940db8d10b3a205dd8470; no voters:
I20260620 14:30:08.563426 20042 raft_consensus.cc:2781] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:08.563520 20042 raft_consensus.cc:482] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:08.563557 20042 raft_consensus.cc:3037] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:08.564288 20042 raft_consensus.cc:504] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34145 } } peers { permanent_uuid: "bd745259718940db8d10b3a205dd8470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38209 } } peers { permanent_uuid: "bc3705e68d444736998efff7987a9f5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36423 } }
I20260620 14:30:08.564446 20042 leader_election.cc:288] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [CANDIDATE]: Term 1 election: Requested vote from peers bd745259718940db8d10b3a205dd8470 (127.0.98.3:38209), bc3705e68d444736998efff7987a9f5f (127.0.98.2:36423)
I20260620 14:30:08.564610 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907f0a81e94246d28cb4e91676cbe09b" candidate_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3705e68d444736998efff7987a9f5f"
I20260620 14:30:08.564690 19845 raft_consensus.cc:3037] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:08.565454 19845 raft_consensus.cc:2454] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a36a5a60a94d9eb01f63ca1fc28de8 in term 1.
I20260620 14:30:08.565585 19976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907f0a81e94246d28cb4e91676cbe09b" candidate_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd745259718940db8d10b3a205dd8470"
I20260620 14:30:08.565636 19650 leader_election.cc:302] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a36a5a60a94d9eb01f63ca1fc28de8, bc3705e68d444736998efff7987a9f5f; no voters:
I20260620 14:30:08.565677 19976 raft_consensus.cc:3037] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:08.565793 20042 raft_consensus.cc:2781] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:08.565961 20042 raft_consensus.cc:686] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [term 1 LEADER]: Becoming Leader. State: Replica: 13a36a5a60a94d9eb01f63ca1fc28de8, State: Running, Role: LEADER
I20260620 14:30:08.566061 20042 consensus_queue.cc:237] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34145 } } peers { permanent_uuid: "bd745259718940db8d10b3a205dd8470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38209 } } peers { permanent_uuid: "bc3705e68d444736998efff7987a9f5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36423 } }
I20260620 14:30:08.566649 19976 raft_consensus.cc:2454] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a36a5a60a94d9eb01f63ca1fc28de8 in term 1.
I20260620 14:30:08.566978 19572 catalog_manager.cc:5495] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13a36a5a60a94d9eb01f63ca1fc28de8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a36a5a60a94d9eb01f63ca1fc28de8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd745259718940db8d10b3a205dd8470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc3705e68d444736998efff7987a9f5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36423 } health_report { overall_health: UNKNOWN } } }
I20260620 14:30:08.574163 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.571957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.0.98.1:34145)
0620 14:30:08.572010 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.572012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.572019 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:30:08.572045 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.572046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.572061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.572117 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:08.572266 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.572705 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.573456 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:08.573628 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:08.573674 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:08.573855 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:08.573945 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:08.574005 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:08.574040 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 14:30:08.574056 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:30:08.576035 19976 raft_consensus.cc:1261] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470 [term 1 FOLLOWER]: Refusing update from remote peer 13a36a5a60a94d9eb01f63ca1fc28de8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:08.576071 19845 raft_consensus.cc:1261] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f [term 1 FOLLOWER]: Refusing update from remote peer 13a36a5a60a94d9eb01f63ca1fc28de8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:08.576416 20046 consensus_queue.cc:1035] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc3705e68d444736998efff7987a9f5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:08.576519 20042 consensus_queue.cc:1035] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd745259718940db8d10b3a205dd8470" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:08.579109 20049 mvcc.cc:205] Tried to move back new op lower bound from 7298931951924789248 to 7298931951887216640. Current Snapshot: MvccSnapshot[applied={T|T < 7298931951924789248}]
I20260620 14:30:08.581827 19571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:08.581926 19571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:08.582718 19571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8406fb8ed7de43a6b9dd3746d96f1082: Sending DeleteTablet for 3 replicas of tablet 907f0a81e94246d28cb4e91676cbe09b
I20260620 14:30:08.583019 19694 tablet_service.cc:1514] Processing DeleteTablet for tablet 907f0a81e94246d28cb4e91676cbe09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:08 UTC) from {username='slave'} at 127.0.0.1:50568
I20260620 14:30:08.583091 19956 tablet_service.cc:1514] Processing DeleteTablet for tablet 907f0a81e94246d28cb4e91676cbe09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:08 UTC) from {username='slave'} at 127.0.0.1:47966
I20260620 14:30:08.583406 20059 tablet_replica.cc:330] T 907f0a81e94246d28cb4e91676cbe09b P 13a36a5a60a94d9eb01f63ca1fc28de8: stopping tablet replica
I20260620 14:30:08.583460 19825 tablet_service.cc:1514] Processing DeleteTablet for tablet 907f0a81e94246d28cb4e91676cbe09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:08 UTC) from {username='slave'} at 127.0.0.1:43098
I20260620 14:30:08.583660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 19632
I20260620 14:30:08.584612 20060 tablet_replica.cc:330] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470: stopping tablet replica
I20260620 14:30:08.584753 20060 raft_consensus.cc:2229] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:08.584887 20060 raft_consensus.cc:2258] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:08.585074 20061 tablet_replica.cc:330] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f: stopping tablet replica
I20260620 14:30:08.585201 20061 raft_consensus.cc:2229] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:08.585300 20061 raft_consensus.cc:2258] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:08.585772 20061 ts_tablet_manager.cc:1905] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:08.586700 20060 ts_tablet_manager.cc:1905] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:08.587632 20061 ts_tablet_manager.cc:1918] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:08.587719 20061 log.cc:1196] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-1/wal/wals/907f0a81e94246d28cb4e91676cbe09b
I20260620 14:30:08.587992 20061 ts_tablet_manager.cc:1939] T 907f0a81e94246d28cb4e91676cbe09b P bc3705e68d444736998efff7987a9f5f: Deleting consensus metadata
I20260620 14:30:08.588438 20060 ts_tablet_manager.cc:1918] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:08.588512 20060 log.cc:1196] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470: Deleting WAL directory at /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965454096380-392-0/minicluster-data/ts-2/wal/wals/907f0a81e94246d28cb4e91676cbe09b
I20260620 14:30:08.588665 19559 catalog_manager.cc:4841] TS bc3705e68d444736998efff7987a9f5f (127.0.98.2:36423): tablet 907f0a81e94246d28cb4e91676cbe09b (table test-table [id=0b379243902c4b2a9e7a526dbf5408ab]) successfully deleted
I20260620 14:30:08.588811 20060 ts_tablet_manager.cc:1939] T 907f0a81e94246d28cb4e91676cbe09b P bd745259718940db8d10b3a205dd8470: Deleting consensus metadata
I20260620 14:30:08.589249 19558 catalog_manager.cc:4841] TS bd745259718940db8d10b3a205dd8470 (127.0.98.3:38209): tablet 907f0a81e94246d28cb4e91676cbe09b (table test-table [id=0b379243902c4b2a9e7a526dbf5408ab]) successfully deleted
W20260620 14:30:08.589402 19558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34145 (error 108)
I20260620 14:30:08.589627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 19763
I20260620 14:30:08.590282 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:08.589713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34145 (local address 127.0.0.1:50586)
0620 14:30:08.590210 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:08.590253 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477}
W20260620 14:30:08.590437 19558 catalog_manager.cc:4584] TS 13a36a5a60a94d9eb01f63ca1fc28de8 (127.0.98.1:34145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 907f0a81e94246d28cb4e91676cbe09b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34145: connect: Connection refused (error 111)
I20260620 14:30:08.594907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 19894
I20260620 14:30:08.600016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 19541
2026-06-20T14:30:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (680 ms)
[----------] 6 tests from EncryptionPolicyTest (8354 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:08 dist-test-slave-hfc3 krb5kdc[20065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:08 dist-test-slave-hfc3 krb5kdc[20065](info): set up 2 sockets
Jun 20 14:30:08 dist-test-slave-hfc3 krb5kdc[20065](info): commencing operation
krb5kdc: starting...
W20260620 14:30:09.650516 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:09 dist-test-slave-hfc3 krb5kdc[20065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:30:09.771143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:09.864367 20078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:09.864653 20078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:09.864682 20078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:09.865881 20078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:09.865922 20078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:09.865937 20078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:09.865949 20078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:09.865962 20078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:09.867589 20078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:09.867810 20078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:09.868000 20078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:09.873145 20083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:09.873152 20084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:09.873215 20086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:09.873456 20078 server_base.cc:1034] running on GCE node
I20260620 14:30:09.873749 20078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:09.874059 20078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:09.875247 20078 hybrid_clock.cc:648] HybridClock initialized: now 1781965809875226 us; error 53 us; skew 500 ppm
Jun 20 14:30:09 dist-test-slave-hfc3 krb5kdc[20065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:09.877812 20078 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:30:09.878278 20078 webserver.cc:460] Webserver started at http://127.0.0.1:39583/ using document root <none> and password file <none>
I20260620 14:30:09.878541 20078 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:09.878589 20078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:09.878710 20078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:09.879732 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "018584e580364ddebd3d2c78fc7deedd"
format_stamp: "Formatted at 2026-06-20 14:30:09 on dist-test-slave-hfc3"
server_key: "c30564f4c83abb431869bac02339079e"
server_key_iv: "32339dfbf176ec548c202c941849dcdb"
server_key_version: "encryptionkey@0"
I20260620 14:30:09.880039 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "018584e580364ddebd3d2c78fc7deedd"
format_stamp: "Formatted at 2026-06-20 14:30:09 on dist-test-slave-hfc3"
server_key: "c30564f4c83abb431869bac02339079e"
server_key_iv: "32339dfbf176ec548c202c941849dcdb"
server_key_version: "encryptionkey@0"
I20260620 14:30:09.881424 20078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:09.882234 20093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:09.882457 20078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:09.882550 20078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "018584e580364ddebd3d2c78fc7deedd"
format_stamp: "Formatted at 2026-06-20 14:30:09 on dist-test-slave-hfc3"
server_key: "c30564f4c83abb431869bac02339079e"
server_key_iv: "32339dfbf176ec548c202c941849dcdb"
server_key_version: "encryptionkey@0"
I20260620 14:30:09.882609 20078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:09.897104 20078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:09.899305 20078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:09.899425 20078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:09.903403 20078 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38855
I20260620 14:30:09.903436 20145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38855 every 8 connection(s)
I20260620 14:30:09.903811 20078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:09.904781 20146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:09.906920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 20078
I20260620 14:30:09.907042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:09.907279 392 external_mini_cluster.cc:1428] Setting key e92f4edee2109169324390ea09132db4
I20260620 14:30:09.907369 20146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd: Bootstrap starting.
I20260620 14:30:09.908030 20146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:09.908349 20146 log.cc:824] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:09.909121 20146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd: No bootstrap required, opened a new log
I20260620 14:30:09.911365 20146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018584e580364ddebd3d2c78fc7deedd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260620 14:30:09.911536 20146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:09.911567 20146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 018584e580364ddebd3d2c78fc7deedd, State: Initialized, Role: FOLLOWER
I20260620 14:30:09.911688 20146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018584e580364ddebd3d2c78fc7deedd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260620 14:30:09.911794 20146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:09.911859 20146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:09.911911 20146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:09.912552 20146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018584e580364ddebd3d2c78fc7deedd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260620 14:30:09.912671 20146 leader_election.cc:302] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 018584e580364ddebd3d2c78fc7deedd; no voters:
I20260620 14:30:09.912865 20146 leader_election.cc:288] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:09.912894 20151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:09.913012 20151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [term 1 LEADER]: Becoming Leader. State: Replica: 018584e580364ddebd3d2c78fc7deedd, State: Running, Role: LEADER
I20260620 14:30:09.913146 20146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:09.913130 20151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018584e580364ddebd3d2c78fc7deedd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260620 14:30:09.913455 20151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "018584e580364ddebd3d2c78fc7deedd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018584e580364ddebd3d2c78fc7deedd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } } }
I20260620 14:30:09.913519 20151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:09.913637 20151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 018584e580364ddebd3d2c78fc7deedd. Latest consensus state: current_term: 1 leader_uuid: "018584e580364ddebd3d2c78fc7deedd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018584e580364ddebd3d2c78fc7deedd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } } }
I20260620 14:30:09.913695 20151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd [sys.catalog]: This master's current role is: LEADER
Jun 20 14:30:09 dist-test-slave-hfc3 krb5kdc[20065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260620 14:30:09.915073 20167 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:09.915136 20167 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:09.915190 20156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:09.915377 20156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:09.917104 20156 catalog_manager.cc:1347] Generated new cluster ID: c40268d885df4a349937a47a690cd04d
I20260620 14:30:09.917158 20156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:09.919445 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:09.907834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58239 (local address 127.0.0.1:38855)
0620 14:30:09.908031 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:09.908040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:09.908186 (+ 146us) server_negotiation.cc:407] Connection header received
0620 14:30:09.908681 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:09.908688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:09.908737 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:09.908884 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:09.909695 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.910676 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.911796 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.912321 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.912408 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:09.915518 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:09.915522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:09.915523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:09.915534 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:09.917673 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:09.918150 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:09.918153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:09.918153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:09.918267 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:09.918533 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:09.918534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:09.918534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:09.918824 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:09.918972 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:09.919123 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:30:09.919224 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260620 14:30:09.920558 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:09.934273 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:09.922611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60447 (local address 127.0.0.1:38855)
0620 14:30:09.922645 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:09.922646 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:09.926531 (+ 3885us) server_negotiation.cc:407] Connection header received
0620 14:30:09.926603 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:09.926604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:09.926614 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:09.926665 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:09.927346 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.928107 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.929336 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.929686 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.929760 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:09.931360 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:09.931361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:09.931361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:09.931369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:09.932984 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:09.933440 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:09.933441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:09.933442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:09.933500 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:09.933736 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:09.933736 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:09.933737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:09.933859 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:09.933949 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:09.934111 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:30:09.934175 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:30:09.940653 20156 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:09.941411 20156 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:09.951897 20156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 018584e580364ddebd3d2c78fc7deedd: Generated new TSK 0
I20260620 14:30:09.952152 20156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:09.961730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 20078
W20260620 14:30:09.966759 20063 connection.cc:535] client connection to 127.0.0.1:38855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38855): Connection reset by peer (error 104)
2026-06-20T14:30:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:10 dist-test-slave-hfc3 krb5kdc[20177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:10 dist-test-slave-hfc3 krb5kdc[20177](info): set up 2 sockets
Jun 20 14:30:10 dist-test-slave-hfc3 krb5kdc[20177](info): commencing operation
krb5kdc: starting...
W20260620 14:30:10.990499 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:11 dist-test-slave-hfc3 krb5kdc[20177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:30:11.113119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:11.197007 20190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:11.197222 20190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:11.197247 20190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:11.198364 20190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:11.198401 20190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:11.198415 20190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:11.198428 20190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:11.198441 20190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:11.199874 20190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34011
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:11.200074 20190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:11.200248 20190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:11.205188 20198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:11.205171 20196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:11.205171 20195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:11.205294 20190 server_base.cc:1034] running on GCE node
I20260620 14:30:11.205684 20190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:11.205991 20190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:11.207161 20190 hybrid_clock.cc:648] HybridClock initialized: now 1781965811207149 us; error 31 us; skew 500 ppm
Jun 20 14:30:11 dist-test-slave-hfc3 krb5kdc[20177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:11.215039 20190 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:30:11.215565 20190 webserver.cc:460] Webserver started at http://127.0.0.1:33117/ using document root <none> and password file <none>
I20260620 14:30:11.215785 20190 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:11.215828 20190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:11.215920 20190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:11.216982 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "6ef0a66ca03744a69857a7b089530c5b"
format_stamp: "Formatted at 2026-06-20 14:30:11 on dist-test-slave-hfc3"
server_key: "519a7a54399427ae274c7e0d6df979b8"
server_key_iv: "3a1075a114ec88f32ab9347f746a85b2"
server_key_version: "encryptionkey@0"
I20260620 14:30:11.217293 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ef0a66ca03744a69857a7b089530c5b"
format_stamp: "Formatted at 2026-06-20 14:30:11 on dist-test-slave-hfc3"
server_key: "519a7a54399427ae274c7e0d6df979b8"
server_key_iv: "3a1075a114ec88f32ab9347f746a85b2"
server_key_version: "encryptionkey@0"
I20260620 14:30:11.218709 20190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:30:11.219520 20205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:11.219722 20190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:11.219810 20190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "6ef0a66ca03744a69857a7b089530c5b"
format_stamp: "Formatted at 2026-06-20 14:30:11 on dist-test-slave-hfc3"
server_key: "519a7a54399427ae274c7e0d6df979b8"
server_key_iv: "3a1075a114ec88f32ab9347f746a85b2"
server_key_version: "encryptionkey@0"
I20260620 14:30:11.219872 20190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:11.231289 20190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:11.233696 20190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:11.233816 20190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:11.237900 20190 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34011
I20260620 14:30:11.237957 20257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34011 every 8 connection(s)
I20260620 14:30:11.238399 20190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:11.239356 20258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:11.241683 20258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b: Bootstrap starting.
I20260620 14:30:11.242291 20258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:11.242635 20258 log.cc:824] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:11.243372 20258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b: No bootstrap required, opened a new log
I20260620 14:30:11.245021 20258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef0a66ca03744a69857a7b089530c5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34011 } }
I20260620 14:30:11.245193 20258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:11.245235 20258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef0a66ca03744a69857a7b089530c5b, State: Initialized, Role: FOLLOWER
I20260620 14:30:11.245428 20258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef0a66ca03744a69857a7b089530c5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34011 } }
I20260620 14:30:11.245535 20258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:11.245584 20258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:11.245635 20258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:11.246197 20258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef0a66ca03744a69857a7b089530c5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34011 } }
I20260620 14:30:11.246321 20258 leader_election.cc:302] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ef0a66ca03744a69857a7b089530c5b; no voters:
I20260620 14:30:11.246541 20258 leader_election.cc:288] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:11.246593 20261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:11.246822 20261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [term 1 LEADER]: Becoming Leader. State: Replica: 6ef0a66ca03744a69857a7b089530c5b, State: Running, Role: LEADER
I20260620 14:30:11.246845 20258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:11.246922 20261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef0a66ca03744a69857a7b089530c5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34011 } }
I20260620 14:30:11.247362 20262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ef0a66ca03744a69857a7b089530c5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef0a66ca03744a69857a7b089530c5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34011 } } }
I20260620 14:30:11.247335 20263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef0a66ca03744a69857a7b089530c5b. Latest consensus state: current_term: 1 leader_uuid: "6ef0a66ca03744a69857a7b089530c5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ef0a66ca03744a69857a7b089530c5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34011 } } }
I20260620 14:30:11.247430 20262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:11.247438 20263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:11.247653 20265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:11.248250 20265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:11.248636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 20190
I20260620 14:30:11.248751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:11.248960 392 external_mini_cluster.cc:1428] Setting key 7bb0507e13be0d840d66542747d35392
I20260620 14:30:11.249987 20265 catalog_manager.cc:1347] Generated new cluster ID: 2dfa2cfa9c7d49fa85e45aa1d6aae9b7
I20260620 14:30:11.250046 20265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:11.254062 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:11.249457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42375 (local address 127.0.0.1:34011)
0620 14:30:11.249642 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.249652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.249690 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:30:11.249832 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.249840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.249898 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.250052 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.250290 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:11.251257 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:11.252083 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:11.252519 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:11.252654 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:11.253423 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:11.253426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:11.253427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:11.253612 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:11.253644 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:11.253655 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:11.253759 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:11.253838 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 14:30:11.253867 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260620 14:30:11.255065 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:11.261458 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:11.256506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42807 (local address 127.0.0.1:34011)
0620 14:30:11.256546 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.256548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.258526 (+ 1978us) server_negotiation.cc:407] Connection header received
0620 14:30:11.258605 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.258606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.258617 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.258675 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.258818 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:11.259348 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:11.260091 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:11.260331 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:11.260419 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:11.261170 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:11.261171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:11.261172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:11.261176 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:11.261192 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:11.261197 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:11.261275 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:11.261337 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:30:11.261351 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:30:11.265692 20265 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:11.266230 20265 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:11.276628 20265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ef0a66ca03744a69857a7b089530c5b: Generated new TSK 0
I20260620 14:30:11.276844 20265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:11.282274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 20190
W20260620 14:30:11.286945 20175 connection.cc:535] client connection to 127.0.0.1:34011 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34011): Connection reset by peer (error 104)
2026-06-20T14:30:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:11 dist-test-slave-hfc3 krb5kdc[20289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:11 dist-test-slave-hfc3 krb5kdc[20289](info): set up 2 sockets
Jun 20 14:30:11 dist-test-slave-hfc3 krb5kdc[20289](info): commencing operation
krb5kdc: starting...
W20260620 14:30:12.326589 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:12 dist-test-slave-hfc3 krb5kdc[20289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:30:12.450234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:12.533288 20302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:12.533548 20302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:12.533591 20302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:12.534847 20302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:12.534922 20302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:12.534947 20302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:12.534977 20302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:12.535007 20302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:12.536561 20302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:12.536866 20302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:12.537110 20302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:12.542251 20310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:12.542255 20308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:12.542310 20302 server_base.cc:1034] running on GCE node
W20260620 14:30:12.542255 20307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:12.542822 20302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:12.543179 20302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:12.544382 20302 hybrid_clock.cc:648] HybridClock initialized: now 1781965812544367 us; error 33 us; skew 500 ppm
Jun 20 14:30:12 dist-test-slave-hfc3 krb5kdc[20289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:12.547227 20302 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:30:12.547874 20302 webserver.cc:460] Webserver started at http://127.0.0.1:38911/ using document root <none> and password file <none>
I20260620 14:30:12.548133 20302 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:12.548207 20302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:12.548363 20302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:12.549561 20302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "59255fedf5da44bcaeb34c68bac3d048"
format_stamp: "Formatted at 2026-06-20 14:30:12 on dist-test-slave-hfc3"
server_key: "59ef58a9be460bf925df105291e96b5b"
server_key_iv: "fa0e6c75a2803c79dfc7ca15fd35cfe7"
server_key_version: "encryptionkey@0"
I20260620 14:30:12.549949 20302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "59255fedf5da44bcaeb34c68bac3d048"
format_stamp: "Formatted at 2026-06-20 14:30:12 on dist-test-slave-hfc3"
server_key: "59ef58a9be460bf925df105291e96b5b"
server_key_iv: "fa0e6c75a2803c79dfc7ca15fd35cfe7"
server_key_version: "encryptionkey@0"
I20260620 14:30:12.551504 20302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:30:12.552402 20317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:12.552667 20302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:12.552788 20302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "59255fedf5da44bcaeb34c68bac3d048"
format_stamp: "Formatted at 2026-06-20 14:30:12 on dist-test-slave-hfc3"
server_key: "59ef58a9be460bf925df105291e96b5b"
server_key_iv: "fa0e6c75a2803c79dfc7ca15fd35cfe7"
server_key_version: "encryptionkey@0"
I20260620 14:30:12.552881 20302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:12.556532 20302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:12.559202 20302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:12.559381 20302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:12.563638 20302 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45799
I20260620 14:30:12.563676 20369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45799 every 8 connection(s)
I20260620 14:30:12.564056 20302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:12.565052 20370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:12.565382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 20302
I20260620 14:30:12.565510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:12.565757 392 external_mini_cluster.cc:1428] Setting key 73c57283946c21d30ff53a78bbc34171
I20260620 14:30:12.568354 20370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048: Bootstrap starting.
I20260620 14:30:12.568609 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:12.566372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42895 (local address 127.0.0.1:45799)
0620 14:30:12.566605 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:12.566614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:12.566651 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:30:12.567081 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:12.567088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:12.567119 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:12.567281 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:12.568016 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:12.568019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:12.568020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:12.568024 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:12.568071 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:12.568088 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:12.568173 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:12.568254 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:30:12.568387 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
I20260620 14:30:12.569161 20370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:12.569491 20370 log.cc:824] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:12.569882 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:12.570283 20370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048: No bootstrap required, opened a new log
I20260620 14:30:12.572419 20370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59255fedf5da44bcaeb34c68bac3d048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } }
I20260620 14:30:12.572633 20370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:12.572691 20370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59255fedf5da44bcaeb34c68bac3d048, State: Initialized, Role: FOLLOWER
I20260620 14:30:12.572732 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:12.571546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39947 (local address 127.0.0.1:45799)
0620 14:30:12.571604 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:12.571606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:12.571691 (+ 85us) server_negotiation.cc:407] Connection header received
0620 14:30:12.571732 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:12.571733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:12.571738 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:12.571812 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:12.572473 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:12.572474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:12.572475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:12.572479 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:12.572493 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:12.572497 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:12.572551 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:12.572603 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:30:12.572616 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:30:12.572824 20370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59255fedf5da44bcaeb34c68bac3d048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } }
I20260620 14:30:12.572933 20370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:12.572976 20370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:12.573026 20370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:12.573658 20370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59255fedf5da44bcaeb34c68bac3d048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } }
I20260620 14:30:12.573778 20370 leader_election.cc:302] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59255fedf5da44bcaeb34c68bac3d048; no voters:
I20260620 14:30:12.573982 20370 leader_election.cc:288] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:12.574074 20380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:12.574245 20380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [term 1 LEADER]: Becoming Leader. State: Replica: 59255fedf5da44bcaeb34c68bac3d048, State: Running, Role: LEADER
I20260620 14:30:12.574265 20370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:12.574368 20380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59255fedf5da44bcaeb34c68bac3d048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } }
I20260620 14:30:12.574707 20382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59255fedf5da44bcaeb34c68bac3d048. Latest consensus state: current_term: 1 leader_uuid: "59255fedf5da44bcaeb34c68bac3d048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59255fedf5da44bcaeb34c68bac3d048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } } }
I20260620 14:30:12.574738 20381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59255fedf5da44bcaeb34c68bac3d048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59255fedf5da44bcaeb34c68bac3d048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } } }
I20260620 14:30:12.574806 20382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:12.574824 20381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:12.575384 20387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:12.575918 20387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:12.577587 20387 catalog_manager.cc:1347] Generated new cluster ID: 87774ff6d94648a196466f468042914c
I20260620 14:30:12.577631 20387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:12.592890 20387 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:12.593129 20387 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:12.599368 20387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59255fedf5da44bcaeb34c68bac3d048: Generated new TSK 0
I20260620 14:30:12.599571 20387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:12.602188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 20302
2026-06-20T14:30:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:12 dist-test-slave-hfc3 krb5kdc[20401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:12 dist-test-slave-hfc3 krb5kdc[20401](info): set up 2 sockets
Jun 20 14:30:12 dist-test-slave-hfc3 krb5kdc[20401](info): commencing operation
krb5kdc: starting...
W20260620 14:30:13.678884 392 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:13 dist-test-slave-hfc3 krb5kdc[20401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:30:13.799603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:13.881628 20414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:13.881883 20414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:13.881922 20414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:13.883189 20414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:13.883260 20414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:13.883284 20414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:13.883313 20414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:13.883344 20414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:13.884801 20414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:13.885082 20414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:13.885311 20414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:13.890060 20420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:13.890069 20419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:13.890184 20422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:13.890265 20414 server_base.cc:1034] running on GCE node
I20260620 14:30:13.890527 20414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:13.890832 20414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:13.892004 20414 hybrid_clock.cc:648] HybridClock initialized: now 1781965813891989 us; error 45 us; skew 500 ppm
Jun 20 14:30:13 dist-test-slave-hfc3 krb5kdc[20401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:13.894639 20414 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:30:13.895124 20414 webserver.cc:460] Webserver started at http://127.0.0.1:39661/ using document root <none> and password file <none>
I20260620 14:30:13.895350 20414 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:13.895397 20414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:13.895527 20414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:13.896596 20414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "80dd8bb93c1f4d529fad129b136087c8"
format_stamp: "Formatted at 2026-06-20 14:30:13 on dist-test-slave-hfc3"
server_key: "7e133c00a2069864457de9e228d0ea37"
server_key_iv: "bdccff47abc5d9f4b0e290bf33579b0d"
server_key_version: "encryptionkey@0"
I20260620 14:30:13.896966 20414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "80dd8bb93c1f4d529fad129b136087c8"
format_stamp: "Formatted at 2026-06-20 14:30:13 on dist-test-slave-hfc3"
server_key: "7e133c00a2069864457de9e228d0ea37"
server_key_iv: "bdccff47abc5d9f4b0e290bf33579b0d"
server_key_version: "encryptionkey@0"
I20260620 14:30:13.898294 20414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:30:13.899116 20429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:13.899298 20414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:13.899386 20414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "80dd8bb93c1f4d529fad129b136087c8"
format_stamp: "Formatted at 2026-06-20 14:30:13 on dist-test-slave-hfc3"
server_key: "7e133c00a2069864457de9e228d0ea37"
server_key_iv: "bdccff47abc5d9f4b0e290bf33579b0d"
server_key_version: "encryptionkey@0"
I20260620 14:30:13.899441 20414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:13.917472 20414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:13.919761 20414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:13.919881 20414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:13.924021 20414 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42143
I20260620 14:30:13.924062 20481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42143 every 8 connection(s)
I20260620 14:30:13.924466 20414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:13.924925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 20414
I20260620 14:30:13.925071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:13.925325 392 external_mini_cluster.cc:1428] Setting key 5439162a882cb24e6f57c3c802fac01d
I20260620 14:30:13.925556 20482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:13.928390 20482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8: Bootstrap starting.
I20260620 14:30:13.929003 20482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:13.929330 20482 log.cc:824] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:13.930110 20482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8: No bootstrap required, opened a new log
Jun 20 14:30:13 dist-test-slave-hfc3 krb5kdc[20401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:30:13.932343 20482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80dd8bb93c1f4d529fad129b136087c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42143 } }
I20260620 14:30:13.932529 20482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:13.932606 20482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80dd8bb93c1f4d529fad129b136087c8, State: Initialized, Role: FOLLOWER
I20260620 14:30:13.932808 20482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80dd8bb93c1f4d529fad129b136087c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42143 } }
I20260620 14:30:13.932878 20482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:13.932935 20482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:13.932989 20482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:13.933672 20482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80dd8bb93c1f4d529fad129b136087c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42143 } }
I20260620 14:30:13.933816 20482 leader_election.cc:302] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80dd8bb93c1f4d529fad129b136087c8; no voters:
I20260620 14:30:13.934019 20482 leader_election.cc:288] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:13.934054 20487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:13.934187 20487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [term 1 LEADER]: Becoming Leader. State: Replica: 80dd8bb93c1f4d529fad129b136087c8, State: Running, Role: LEADER
I20260620 14:30:13.934270 20487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80dd8bb93c1f4d529fad129b136087c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42143 } }
I20260620 14:30:13.934315 20482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:13.934620 20487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80dd8bb93c1f4d529fad129b136087c8. Latest consensus state: current_term: 1 leader_uuid: "80dd8bb93c1f4d529fad129b136087c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80dd8bb93c1f4d529fad129b136087c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42143 } } }
I20260620 14:30:13.934698 20487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:13.934633 20489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80dd8bb93c1f4d529fad129b136087c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80dd8bb93c1f4d529fad129b136087c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42143 } } }
I20260620 14:30:13.934876 20489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:13.935479 20492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:13.936115 20492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:13.936887 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:13.925863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47411 (local address 127.0.0.1:42143)
0620 14:30:13.926079 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:13.926129 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:13.926155 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:30:13.926653 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:13.926661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:13.926709 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:13.926857 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:13.927490 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:13.928363 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:13.929166 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:13.929736 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:13.929878 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:13.932563 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:13.932568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:13.932568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:13.932582 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:13.934707 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:13.935141 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:13.935143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:13.935144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:13.935220 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:13.935444 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:13.935444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:13.935445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:13.935818 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:13.936456 (+ 638us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:13.936531 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:30:13.936674 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
I20260620 14:30:13.937799 20492 catalog_manager.cc:1347] Generated new cluster ID: 01c0e49205c24fadb11981146b2096f0
I20260620 14:30:13.937851 20492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:13.937923 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:13.949683 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:13.939415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41229 (local address 127.0.0.1:42143)
0620 14:30:13.939447 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:13.939448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:13.942494 (+ 3046us) server_negotiation.cc:407] Connection header received
0620 14:30:13.942596 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:13.942597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:13.942608 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:13.942682 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:13.943297 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:13.944073 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:13.945050 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:13.945427 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:13.945501 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:13.946873 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:13.946874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:13.946875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:13.946882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:13.948536 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:13.948914 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:13.948915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:13.948915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:13.948977 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:13.949173 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:13.949174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:13.949174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:13.949308 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:13.949402 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:13.949506 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 14:30:13.949579 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:30:13.951062 20492 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:13.951769 20492 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:13.970129 20492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80dd8bb93c1f4d529fad129b136087c8: Generated new TSK 0
I20260620 14:30:13.970412 20492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:13.978117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 20414
2026-06-20T14:30:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:14 dist-test-slave-hfc3 krb5kdc[20513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:14 dist-test-slave-hfc3 krb5kdc[20513](info): set up 2 sockets
Jun 20 14:30:14 dist-test-slave-hfc3 krb5kdc[20513](info): commencing operation
krb5kdc: starting...
W20260620 14:30:15.014634 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:15 dist-test-slave-hfc3 krb5kdc[20513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:30:15.139335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:15.221410 20526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:15.221655 20526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:15.221689 20526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:15.222870 20526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:15.222944 20526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:15.222967 20526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:15.222988 20526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:15.223019 20526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:15.224452 20526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:15.224735 20526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:15.224963 20526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:15.229833 20534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:15.229816 20532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:15.229817 20531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:15.230136 20526 server_base.cc:1034] running on GCE node
I20260620 14:30:15.230464 20526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:15.230760 20526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:15.231927 20526 hybrid_clock.cc:648] HybridClock initialized: now 1781965815231914 us; error 28 us; skew 500 ppm
Jun 20 14:30:15 dist-test-slave-hfc3 krb5kdc[20513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:15.234653 20526 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:30:15.235103 20526 webserver.cc:460] Webserver started at http://127.0.0.1:35803/ using document root <none> and password file <none>
I20260620 14:30:15.235335 20526 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:15.235404 20526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:15.235564 20526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:15.236699 20526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "25acca54ecb049fc820b449046ae0a60"
format_stamp: "Formatted at 2026-06-20 14:30:15 on dist-test-slave-hfc3"
server_key: "24e63b51fcbc42e293e3c86e903abdee"
server_key_iv: "9fb835a5493b037aa2d65ac4fce923dc"
server_key_version: "encryptionkey@0"
I20260620 14:30:15.237089 20526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "25acca54ecb049fc820b449046ae0a60"
format_stamp: "Formatted at 2026-06-20 14:30:15 on dist-test-slave-hfc3"
server_key: "24e63b51fcbc42e293e3c86e903abdee"
server_key_iv: "9fb835a5493b037aa2d65ac4fce923dc"
server_key_version: "encryptionkey@0"
I20260620 14:30:15.238521 20526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:15.239236 20541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:15.239403 20526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:15.239508 20526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "25acca54ecb049fc820b449046ae0a60"
format_stamp: "Formatted at 2026-06-20 14:30:15 on dist-test-slave-hfc3"
server_key: "24e63b51fcbc42e293e3c86e903abdee"
server_key_iv: "9fb835a5493b037aa2d65ac4fce923dc"
server_key_version: "encryptionkey@0"
I20260620 14:30:15.239596 20526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:15.256966 20526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:15.259104 20526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:15.259286 20526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:15.263592 20526 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38297
I20260620 14:30:15.263626 20593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38297 every 8 connection(s)
I20260620 14:30:15.264034 20526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:15.264695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 20526
I20260620 14:30:15.264833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:15.265025 392 external_mini_cluster.cc:1428] Setting key 0ecc117bd69668c8b9c9e244ba1097c4
I20260620 14:30:15.265103 20594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:15.268604 20594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60: Bootstrap starting.
I20260620 14:30:15.269295 20594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:15.269588 20594 log.cc:824] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:15.270462 20594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60: No bootstrap required, opened a new log
I20260620 14:30:15.271245 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:15.265547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59467 (local address 127.0.0.1:38297)
0620 14:30:15.265786 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:15.265796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:15.265830 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:30:15.266328 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:15.266338 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:15.266420 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:15.266571 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:15.266765 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:15.268145 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:15.269006 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:15.269638 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:15.269758 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:15.270545 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:15.270550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:15.270550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:15.270767 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:15.270812 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:15.270827 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:15.270890 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:15.270964 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:30:15.271031 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260620 14:30:15.272420 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:15.272923 20594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25acca54ecb049fc820b449046ae0a60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38297 } }
I20260620 14:30:15.273118 20594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:15.273156 20594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25acca54ecb049fc820b449046ae0a60, State: Initialized, Role: FOLLOWER
I20260620 14:30:15.273303 20594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25acca54ecb049fc820b449046ae0a60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38297 } }
I20260620 14:30:15.273381 20594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:15.273411 20594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:15.273464 20594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:15.274227 20594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25acca54ecb049fc820b449046ae0a60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38297 } }
I20260620 14:30:15.274358 20594 leader_election.cc:302] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25acca54ecb049fc820b449046ae0a60; no voters:
I20260620 14:30:15.274549 20594 leader_election.cc:288] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:15.274580 20604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:15.274696 20604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [term 1 LEADER]: Becoming Leader. State: Replica: 25acca54ecb049fc820b449046ae0a60, State: Running, Role: LEADER
I20260620 14:30:15.274788 20604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25acca54ecb049fc820b449046ae0a60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38297 } }
I20260620 14:30:15.274808 20594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:15.275107 20604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25acca54ecb049fc820b449046ae0a60. Latest consensus state: current_term: 1 leader_uuid: "25acca54ecb049fc820b449046ae0a60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25acca54ecb049fc820b449046ae0a60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38297 } } }
I20260620 14:30:15.275178 20604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:15.275328 20605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25acca54ecb049fc820b449046ae0a60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25acca54ecb049fc820b449046ae0a60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38297 } } }
I20260620 14:30:15.275406 20605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:15.275528 20609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:15.276237 20609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:15.277555 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:15.274018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40265 (local address 127.0.0.1:38297)
0620 14:30:15.274075 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:15.274078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:15.274240 (+ 162us) server_negotiation.cc:407] Connection header received
0620 14:30:15.274262 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:15.274263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:15.274274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:15.274325 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:15.274517 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:15.275212 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:15.276238 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:15.276518 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:15.276616 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:15.277319 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:15.277320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:15.277321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:15.277325 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:15.277338 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:15.277342 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:15.277390 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:15.277445 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:30:15.277466 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:30:15.278221 20609 catalog_manager.cc:1347] Generated new cluster ID: 43b274934b30422dabcbd258a9076e21
I20260620 14:30:15.278270 20609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:15.286710 20609 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:15.287464 20609 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:15.294364 20609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25acca54ecb049fc820b449046ae0a60: Generated new TSK 0
I20260620 14:30:15.294564 20609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:15.298187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 20526
2026-06-20T14:30:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:15 dist-test-slave-hfc3 krb5kdc[20625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:15 dist-test-slave-hfc3 krb5kdc[20625](info): set up 2 sockets
Jun 20 14:30:15 dist-test-slave-hfc3 krb5kdc[20625](info): commencing operation
krb5kdc: starting...
W20260620 14:30:16.334586 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:16 dist-test-slave-hfc3 krb5kdc[20625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:30:16.455461 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:16.536315 20638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:16.536567 20638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:16.536607 20638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:16.537806 20638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:16.537881 20638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:16.537907 20638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:16.537940 20638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:16.537964 20638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:16.539539 20638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:16.539845 20638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:16.540088 20638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:16.545151 20644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:16.545171 20646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:16.545244 20643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:16.545359 20638 server_base.cc:1034] running on GCE node
I20260620 14:30:16.545730 20638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:16.546008 20638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:16.547150 20638 hybrid_clock.cc:648] HybridClock initialized: now 1781965816547138 us; error 34 us; skew 500 ppm
Jun 20 14:30:16 dist-test-slave-hfc3 krb5kdc[20625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:16.556311 20638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:30:16.556818 20638 webserver.cc:460] Webserver started at http://127.0.0.1:41459/ using document root <none> and password file <none>
I20260620 14:30:16.557072 20638 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:16.557121 20638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:16.557308 20638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:16.558525 20638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "09d179ae77804da3949fb58515c72475"
format_stamp: "Formatted at 2026-06-20 14:30:16 on dist-test-slave-hfc3"
server_key: "8f1b879cbf2615eaf765d6d58f692496"
server_key_iv: "a5b30fd9ab58ba2578a58f31f27f542e"
server_key_version: "encryptionkey@0"
I20260620 14:30:16.558907 20638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "09d179ae77804da3949fb58515c72475"
format_stamp: "Formatted at 2026-06-20 14:30:16 on dist-test-slave-hfc3"
server_key: "8f1b879cbf2615eaf765d6d58f692496"
server_key_iv: "a5b30fd9ab58ba2578a58f31f27f542e"
server_key_version: "encryptionkey@0"
I20260620 14:30:16.560329 20638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:30:16.561190 20653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:16.561412 20638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:30:16.561509 20638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "09d179ae77804da3949fb58515c72475"
format_stamp: "Formatted at 2026-06-20 14:30:16 on dist-test-slave-hfc3"
server_key: "8f1b879cbf2615eaf765d6d58f692496"
server_key_iv: "a5b30fd9ab58ba2578a58f31f27f542e"
server_key_version: "encryptionkey@0"
I20260620 14:30:16.561585 20638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:16.565192 20638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:16.567230 20638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:16.567386 20638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:16.571542 20638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35665
I20260620 14:30:16.571626 20705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35665 every 8 connection(s)
I20260620 14:30:16.571943 20638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:16.572975 20706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:16.575554 20706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475: Bootstrap starting.
I20260620 14:30:16.576133 20706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:16.576393 20706 log.cc:824] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:16.577116 20706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475: No bootstrap required, opened a new log
I20260620 14:30:16.578917 20706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d179ae77804da3949fb58515c72475" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260620 14:30:16.579074 20706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:16.579100 20706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d179ae77804da3949fb58515c72475, State: Initialized, Role: FOLLOWER
I20260620 14:30:16.579217 20706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d179ae77804da3949fb58515c72475" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260620 14:30:16.579273 20706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:16.579295 20706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:16.579329 20706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:16.579854 20706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d179ae77804da3949fb58515c72475" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260620 14:30:16.579954 20706 leader_election.cc:302] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09d179ae77804da3949fb58515c72475; no voters:
I20260620 14:30:16.580106 20706 leader_election.cc:288] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:16.580178 20709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:16.580376 20709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [term 1 LEADER]: Becoming Leader. State: Replica: 09d179ae77804da3949fb58515c72475, State: Running, Role: LEADER
I20260620 14:30:16.580374 20706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:16.580673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 20638
I20260620 14:30:16.580658 20709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d179ae77804da3949fb58515c72475" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260620 14:30:16.580806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:16.581003 392 external_mini_cluster.cc:1428] Setting key a531adb6950c3fc0dd4ffcffa5430ebc
I20260620 14:30:16.581192 20710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09d179ae77804da3949fb58515c72475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d179ae77804da3949fb58515c72475" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260620 14:30:16.581297 20710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:16.581830 20717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:16.581808 20711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d179ae77804da3949fb58515c72475. Latest consensus state: current_term: 1 leader_uuid: "09d179ae77804da3949fb58515c72475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d179ae77804da3949fb58515c72475" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260620 14:30:16.582018 20711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:16.582605 20717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:16.584087 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:16.581958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35965 (local address 127.0.0.1:35665)
0620 14:30:16.582274 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:16.582283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:16.582371 (+ 88us) server_negotiation.cc:407] Connection header received
0620 14:30:16.582490 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:16.582498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:16.582528 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:16.582672 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:16.583649 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:16.583652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:16.583652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:16.583655 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:16.583690 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:16.583700 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:16.583787 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:16.583861 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:30:16.583902 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":208,"threads_started":1}
I20260620 14:30:16.584306 20717 catalog_manager.cc:1347] Generated new cluster ID: 4549484059c2407a99506c208ee1399e
I20260620 14:30:16.584358 20717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:16.584965 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:16.591533 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:16.590403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36457 (local address 127.0.0.1:35665)
0620 14:30:16.590454 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:16.590456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:16.590467 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:30:16.590493 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:16.590494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:16.590499 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:16.590566 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:16.591296 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:16.591297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:16.591298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:16.591303 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:16.591318 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:16.591322 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:16.591374 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:16.591428 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:30:16.591444 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:30:16.609915 20717 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:16.610111 20717 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:16.612980 20717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09d179ae77804da3949fb58515c72475: Generated new TSK 0
I20260620 14:30:16.613197 20717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:16.616298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 20638
2026-06-20T14:30:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:16 dist-test-slave-hfc3 krb5kdc[20737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:16 dist-test-slave-hfc3 krb5kdc[20737](info): set up 2 sockets
Jun 20 14:30:16 dist-test-slave-hfc3 krb5kdc[20737](info): commencing operation
krb5kdc: starting...
W20260620 14:30:17.658624 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:17 dist-test-slave-hfc3 krb5kdc[20737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:30:17.777595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:17.859376 20750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:17.859653 20750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:17.859697 20750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:17.860948 20750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:17.861027 20750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:17.861052 20750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:17.861084 20750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:17.861129 20750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:17.862785 20750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:17.863070 20750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:17.863313 20750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:17.868661 20755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:17.868669 20756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:17.868682 20758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:17.868929 20750 server_base.cc:1034] running on GCE node
I20260620 14:30:17.869205 20750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:17.869467 20750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:17.870604 20750 hybrid_clock.cc:648] HybridClock initialized: now 1781965817870597 us; error 39 us; skew 500 ppm
Jun 20 14:30:17 dist-test-slave-hfc3 krb5kdc[20737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:17.873715 20750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:30:17.874212 20750 webserver.cc:460] Webserver started at http://127.0.0.1:32839/ using document root <none> and password file <none>
I20260620 14:30:17.874517 20750 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:17.874593 20750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:17.874755 20750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:17.875855 20750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "150fefcf3c54456aa1eb78791a3ea14f"
format_stamp: "Formatted at 2026-06-20 14:30:17 on dist-test-slave-hfc3"
server_key: "c421a8282a47ea5b9d439550d796ae69"
server_key_iv: "b42751e1b856983a44f0200c0c951c24"
server_key_version: "encryptionkey@0"
I20260620 14:30:17.876256 20750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "150fefcf3c54456aa1eb78791a3ea14f"
format_stamp: "Formatted at 2026-06-20 14:30:17 on dist-test-slave-hfc3"
server_key: "c421a8282a47ea5b9d439550d796ae69"
server_key_iv: "b42751e1b856983a44f0200c0c951c24"
server_key_version: "encryptionkey@0"
I20260620 14:30:17.877682 20750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:17.878564 20765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:17.878748 20750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:17.878849 20750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "150fefcf3c54456aa1eb78791a3ea14f"
format_stamp: "Formatted at 2026-06-20 14:30:17 on dist-test-slave-hfc3"
server_key: "c421a8282a47ea5b9d439550d796ae69"
server_key_iv: "b42751e1b856983a44f0200c0c951c24"
server_key_version: "encryptionkey@0"
I20260620 14:30:17.878933 20750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:17.890543 20750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:17.892566 20750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:17.892755 20750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:17.897050 20750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34593
I20260620 14:30:17.897083 20817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34593 every 8 connection(s)
I20260620 14:30:17.897452 20750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:17.898373 20818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:17.900727 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f: Bootstrap starting.
I20260620 14:30:17.901340 20818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:17.901643 20818 log.cc:824] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:17.902469 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f: No bootstrap required, opened a new log
I20260620 14:30:17.902690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 20750
I20260620 14:30:17.902812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:17.903005 392 external_mini_cluster.cc:1428] Setting key ee0b8202006dc071b769bf7afdbc8443
I20260620 14:30:17.904632 20818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150fefcf3c54456aa1eb78791a3ea14f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } }
I20260620 14:30:17.904808 20818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:17.904848 20818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 150fefcf3c54456aa1eb78791a3ea14f, State: Initialized, Role: FOLLOWER
I20260620 14:30:17.905025 20818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150fefcf3c54456aa1eb78791a3ea14f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } }
I20260620 14:30:17.905126 20818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:17.905160 20818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:17.905215 20818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:17.906273 20818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150fefcf3c54456aa1eb78791a3ea14f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } }
I20260620 14:30:17.906420 20818 leader_election.cc:302] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 150fefcf3c54456aa1eb78791a3ea14f; no voters:
I20260620 14:30:17.906641 20818 leader_election.cc:288] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:17.906667 20823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:17.906780 20823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [term 1 LEADER]: Becoming Leader. State: Replica: 150fefcf3c54456aa1eb78791a3ea14f, State: Running, Role: LEADER
I20260620 14:30:17.906939 20818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:17.906917 20823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150fefcf3c54456aa1eb78791a3ea14f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } }
I20260620 14:30:17.907352 20824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "150fefcf3c54456aa1eb78791a3ea14f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150fefcf3c54456aa1eb78791a3ea14f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } } }
I20260620 14:30:17.907393 20825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 150fefcf3c54456aa1eb78791a3ea14f. Latest consensus state: current_term: 1 leader_uuid: "150fefcf3c54456aa1eb78791a3ea14f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150fefcf3c54456aa1eb78791a3ea14f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } } }
I20260620 14:30:17.907482 20824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:17.907716 20825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:17.907979 20832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:17.908560 20832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:17.910229 20832 catalog_manager.cc:1347] Generated new cluster ID: b7888da55eb3409cb5ad0597cd87a4da
I20260620 14:30:17.910285 20832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:30:17 dist-test-slave-hfc3 krb5kdc[20737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:30:17.915005 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:17.903541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:39405 (local address 127.0.0.1:34593)
0620 14:30:17.903743 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:17.903751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:17.903777 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:30:17.904616 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:17.904624 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:17.904689 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:17.904843 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:17.905656 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:17.906772 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:17.907856 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:17.908486 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:17.908619 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:17.911317 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:17.911321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:17.911322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:17.911333 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:17.913097 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:17.913497 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:17.913499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:17.913500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:17.913613 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:17.913851 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:17.913852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:17.913853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:17.914085 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:17.914718 (+ 633us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:17.914787 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:30:17.914865 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
I20260620 14:30:17.916091 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:17.923834 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:17.917616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:40469 (local address 127.0.0.1:34593)
0620 14:30:17.917666 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:17.917667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:17.917827 (+ 160us) server_negotiation.cc:407] Connection header received
0620 14:30:17.917866 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:17.917867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:17.917877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:17.917922 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:17.918454 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:17.918965 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:17.919694 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:17.919907 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:17.919996 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:17.921171 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:17.921173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:17.921173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:17.921179 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:17.922706 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:17.923060 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:17.923061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:17.923061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:17.923119 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:17.923345 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:17.923346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:17.923347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:17.923454 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:17.923562 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:17.923677 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:30:17.923739 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:30:17.938421 20832 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:17.938992 20832 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:17.942947 20832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 150fefcf3c54456aa1eb78791a3ea14f: Generated new TSK 0
I20260620 14:30:17.943161 20832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:17.946476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 20750
W20260620 14:30:17.950968 20735 connection.cc:535] client connection to 127.0.0.1:34593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34593): Connection reset by peer (error 104)
2026-06-20T14:30:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:17 dist-test-slave-hfc3 krb5kdc[20849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:17 dist-test-slave-hfc3 krb5kdc[20849](info): set up 2 sockets
Jun 20 14:30:17 dist-test-slave-hfc3 krb5kdc[20849](info): commencing operation
krb5kdc: starting...
W20260620 14:30:18.986533 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:19 dist-test-slave-hfc3 krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:30:19.105357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:19.188832 20862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:19.189033 20862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:19.189059 20862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:19.190256 20862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:19.190296 20862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:19.190310 20862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:19.190323 20862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:19.190398 20862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:19.191856 20862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:19.192106 20862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:19.192329 20862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:19.197413 20867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:19.197369 20868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:19.197368 20870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:19.197398 20862 server_base.cc:1034] running on GCE node
I20260620 14:30:19.197901 20862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:19.198222 20862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:19.199393 20862 hybrid_clock.cc:648] HybridClock initialized: now 1781965819199372 us; error 33 us; skew 500 ppm
Jun 20 14:30:19 dist-test-slave-hfc3 krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:19.201872 20862 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:30:19.202397 20862 webserver.cc:460] Webserver started at http://127.0.0.1:36805/ using document root <none> and password file <none>
I20260620 14:30:19.202659 20862 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:19.202728 20862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:19.202888 20862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:19.203970 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "6f817d6886c5448582f7fa11efd53fb2"
format_stamp: "Formatted at 2026-06-20 14:30:19 on dist-test-slave-hfc3"
server_key: "421e91b3c43bfc80a1f13c5b0a472e06"
server_key_iv: "d4191f168add059df3d800dfc814d32f"
server_key_version: "encryptionkey@0"
I20260620 14:30:19.204339 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "6f817d6886c5448582f7fa11efd53fb2"
format_stamp: "Formatted at 2026-06-20 14:30:19 on dist-test-slave-hfc3"
server_key: "421e91b3c43bfc80a1f13c5b0a472e06"
server_key_iv: "d4191f168add059df3d800dfc814d32f"
server_key_version: "encryptionkey@0"
I20260620 14:30:19.205709 20862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:30:19.206568 20877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:19.206760 20862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:30:19.206871 20862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "6f817d6886c5448582f7fa11efd53fb2"
format_stamp: "Formatted at 2026-06-20 14:30:19 on dist-test-slave-hfc3"
server_key: "421e91b3c43bfc80a1f13c5b0a472e06"
server_key_iv: "d4191f168add059df3d800dfc814d32f"
server_key_version: "encryptionkey@0"
I20260620 14:30:19.206959 20862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:19.229658 20862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:19.231802 20862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:19.231968 20862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:19.236208 20862 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37597
I20260620 14:30:19.236238 20929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37597 every 8 connection(s)
I20260620 14:30:19.236614 20862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:19.237488 20930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:19.239738 20930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2: Bootstrap starting.
I20260620 14:30:19.240262 20930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:19.240602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 20862
I20260620 14:30:19.240597 20930 log.cc:824] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:19.240708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:19.240931 392 external_mini_cluster.cc:1428] Setting key 6834bb99ee11d6aa8bdb1671206d042c
I20260620 14:30:19.241402 20930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2: No bootstrap required, opened a new log
I20260620 14:30:19.243299 20930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f817d6886c5448582f7fa11efd53fb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37597 } }
I20260620 14:30:19.243464 20930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:19.243530 20930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f817d6886c5448582f7fa11efd53fb2, State: Initialized, Role: FOLLOWER
I20260620 14:30:19.243707 20930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f817d6886c5448582f7fa11efd53fb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37597 } }
I20260620 14:30:19.243805 20930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:19.243851 20930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:19.243902 20930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:19.244616 20930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f817d6886c5448582f7fa11efd53fb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37597 } }
I20260620 14:30:19.244729 20930 leader_election.cc:302] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f817d6886c5448582f7fa11efd53fb2; no voters:
I20260620 14:30:19.244889 20930 leader_election.cc:288] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:19.245038 20936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:19.245136 20930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:19.245211 20936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f817d6886c5448582f7fa11efd53fb2, State: Running, Role: LEADER
I20260620 14:30:19.245364 20936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f817d6886c5448582f7fa11efd53fb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37597 } }
I20260620 14:30:19.246361 20935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f817d6886c5448582f7fa11efd53fb2. Latest consensus state: current_term: 1 leader_uuid: "6f817d6886c5448582f7fa11efd53fb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f817d6886c5448582f7fa11efd53fb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37597 } } }
I20260620 14:30:19.246622 20935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:19.246543 20937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f817d6886c5448582f7fa11efd53fb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f817d6886c5448582f7fa11efd53fb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37597 } } }
I20260620 14:30:19.246886 20944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:19.246884 20937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:19.247398 20944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:19.249428 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:19.241445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:52789 (local address 127.0.0.1:37597)
0620 14:30:19.241703 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:19.241712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:19.241748 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:30:19.243279 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:19.243286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:19.243342 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:19.243493 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:19.243730 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:19.245597 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:19.246782 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:19.247391 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:19.247501 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:19.248662 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:19.248666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:19.248666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:19.248895 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:19.248933 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:19.248948 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:19.249059 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:19.249178 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 14:30:19.249211 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":75,"threads_started":1}
I20260620 14:30:19.250866 20944 catalog_manager.cc:1347] Generated new cluster ID: 24a6b6e5b9584103a23f9fca0260f4d8
I20260620 14:30:19.250897 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:19.250916 20944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:19.257900 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:19.254076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:45615 (local address 127.0.0.1:37597)
0620 14:30:19.254116 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:19.254118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:19.254260 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:30:19.254324 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:19.254325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:19.254355 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:19.254411 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:19.254586 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:19.255292 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:19.256247 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:19.256787 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:19.256827 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:19.257639 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:19.257640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:19.257641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:19.257645 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:19.257658 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:19.257662 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:19.257693 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:19.257810 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:30:19.257823 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:30:19.264353 20944 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:19.264866 20944 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:19.272655 20944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f817d6886c5448582f7fa11efd53fb2: Generated new TSK 0
I20260620 14:30:19.272895 20944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:19.275290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 20862
W20260620 14:30:19.280360 20847 connection.cc:535] client connection to 127.0.0.1:37597 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37597): Connection reset by peer (error 104)
2026-06-20T14:30:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:19 dist-test-slave-hfc3 krb5kdc[20961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:19 dist-test-slave-hfc3 krb5kdc[20961](info): set up 2 sockets
Jun 20 14:30:19 dist-test-slave-hfc3 krb5kdc[20961](info): commencing operation
krb5kdc: starting...
W20260620 14:30:20.314579 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:20 dist-test-slave-hfc3 krb5kdc[20961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:30:20.436936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:20.520093 20974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:20.520316 20974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:20.520341 20974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:20.521545 20974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:20.521605 20974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:20.521631 20974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:20.521675 20974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:20.521694 20974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:20.523281 20974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:20.523543 20974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:20.523780 20974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:30:20.528910 20974 server_base.cc:1034] running on GCE node
W20260620 14:30:20.528923 20982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:20.528883 20979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:20.528894 20980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:20.529393 20974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:20.529695 20974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:20.530861 20974 hybrid_clock.cc:648] HybridClock initialized: now 1781965820530838 us; error 38 us; skew 500 ppm
Jun 20 14:30:20 dist-test-slave-hfc3 krb5kdc[20961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:20.533442 20974 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:30:20.533856 20974 webserver.cc:460] Webserver started at http://127.0.0.1:41005/ using document root <none> and password file <none>
I20260620 14:30:20.534058 20974 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:20.534101 20974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:20.534197 20974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:20.535327 20974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "f28b438037aa4214845dada3650c95af"
format_stamp: "Formatted at 2026-06-20 14:30:20 on dist-test-slave-hfc3"
server_key: "7f5801f11b8781ef6a89de6c7e6baf4e"
server_key_iv: "7cee73892e992f0ce26cc399b11d8de6"
server_key_version: "encryptionkey@0"
I20260620 14:30:20.535648 20974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "f28b438037aa4214845dada3650c95af"
format_stamp: "Formatted at 2026-06-20 14:30:20 on dist-test-slave-hfc3"
server_key: "7f5801f11b8781ef6a89de6c7e6baf4e"
server_key_iv: "7cee73892e992f0ce26cc399b11d8de6"
server_key_version: "encryptionkey@0"
I20260620 14:30:20.536971 20974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:30:20.537655 20989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:20.537866 20974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:20.537962 20974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "f28b438037aa4214845dada3650c95af"
format_stamp: "Formatted at 2026-06-20 14:30:20 on dist-test-slave-hfc3"
server_key: "7f5801f11b8781ef6a89de6c7e6baf4e"
server_key_iv: "7cee73892e992f0ce26cc399b11d8de6"
server_key_version: "encryptionkey@0"
I20260620 14:30:20.538043 20974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:20.541564 20974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:20.543630 20974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:20.543787 20974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:20.547785 20974 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39945
I20260620 14:30:20.547837 21041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39945 every 8 connection(s)
I20260620 14:30:20.548182 20974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:20.549173 21042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:20.551478 21042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af: Bootstrap starting.
I20260620 14:30:20.552083 21042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:20.552175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 20974
I20260620 14:30:20.552313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:20.552430 21042 log.cc:824] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:20.552515 392 external_mini_cluster.cc:1428] Setting key 55722bdb31adabc540a3f44654418564
I20260620 14:30:20.553347 21042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af: No bootstrap required, opened a new log
I20260620 14:30:20.555152 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:20.553075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:39215 (local address 127.0.0.1:39945)
0620 14:30:20.553258 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:20.553273 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:20.553298 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:30:20.553792 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:20.553800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:20.553828 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:20.553947 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:20.554694 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:20.554697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:20.554697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:20.554701 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:20.554739 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:20.554749 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:20.554817 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:20.554893 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 14:30:20.554949 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260620 14:30:20.556099 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:20.556181 21042 raft_consensus.cc:348] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28b438037aa4214845dada3650c95af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260620 14:30:20.556349 21042 raft_consensus.cc:374] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:20.556380 21042 raft_consensus.cc:729] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28b438037aa4214845dada3650c95af, State: Initialized, Role: FOLLOWER
I20260620 14:30:20.556494 21042 consensus_queue.cc:260] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28b438037aa4214845dada3650c95af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260620 14:30:20.556583 21042 raft_consensus.cc:388] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:20.556618 21042 raft_consensus.cc:482] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:20.556672 21042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:20.557554 21042 raft_consensus.cc:504] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28b438037aa4214845dada3650c95af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260620 14:30:20.557690 21042 leader_election.cc:302] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f28b438037aa4214845dada3650c95af; no voters:
I20260620 14:30:20.557904 21042 leader_election.cc:288] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:20.557960 21052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:20.558139 21052 raft_consensus.cc:686] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [term 1 LEADER]: Becoming Leader. State: Replica: f28b438037aa4214845dada3650c95af, State: Running, Role: LEADER
I20260620 14:30:20.558171 21042 sys_catalog.cc:564] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:20.558290 21052 consensus_queue.cc:237] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28b438037aa4214845dada3650c95af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260620 14:30:20.559335 21053 sys_catalog.cc:455] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f28b438037aa4214845dada3650c95af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28b438037aa4214845dada3650c95af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } } }
I20260620 14:30:20.559433 21053 sys_catalog.cc:458] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:20.559765 21054 sys_catalog.cc:455] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [sys.catalog]: SysCatalogTable state changed. Reason: New leader f28b438037aa4214845dada3650c95af. Latest consensus state: current_term: 1 leader_uuid: "f28b438037aa4214845dada3650c95af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28b438037aa4214845dada3650c95af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } } }
I20260620 14:30:20.559861 21054 sys_catalog.cc:458] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af [sys.catalog]: This master's current role is: LEADER
W20260620 14:30:20.560021 21067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 14:30:20.560087 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:20.557684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:50485 (local address 127.0.0.1:39945)
0620 14:30:20.557720 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:20.557722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:20.558325 (+ 603us) server_negotiation.cc:407] Connection header received
0620 14:30:20.558445 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:20.558446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:20.558452 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:20.558495 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:20.559762 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:20.559763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:20.559764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:20.559768 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:20.559784 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:20.559789 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:20.559837 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:20.559973 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 14:30:20.559986 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:30:20.560094 21067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:20.560222 21068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:20.560580 21068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:20.562016 21068 catalog_manager.cc:1347] Generated new cluster ID: 7c723fe137e54117aca3748a1e6b63b0
I20260620 14:30:20.562057 21068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:20.584447 21068 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:20.584652 21068 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:20.591409 21068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f28b438037aa4214845dada3650c95af: Generated new TSK 0
I20260620 14:30:20.591612 21068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:20.599699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 20974
2026-06-20T14:30:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:20 dist-test-slave-hfc3 krb5kdc[21073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:20 dist-test-slave-hfc3 krb5kdc[21073](info): set up 2 sockets
Jun 20 14:30:20 dist-test-slave-hfc3 krb5kdc[21073](info): commencing operation
krb5kdc: starting...
W20260620 14:30:21.634593 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:21 dist-test-slave-hfc3 krb5kdc[21073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:30:21.753813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:21.861562 21086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:21.861812 21086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:21.861840 21086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:21.863273 21086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:21.863327 21086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:21.863343 21086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:21.863358 21086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:21.863372 21086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:21.865002 21086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:21.865231 21086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:21.865434 21086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:21.870563 21091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:21.870538 21094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:21.870558 21092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:21.870714 21086 server_base.cc:1034] running on GCE node
I20260620 14:30:21.871120 21086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:21.871446 21086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:21.872617 21086 hybrid_clock.cc:648] HybridClock initialized: now 1781965821872597 us; error 43 us; skew 500 ppm
Jun 20 14:30:21 dist-test-slave-hfc3 krb5kdc[21073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:21.875504 21086 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:30:21.876039 21086 webserver.cc:460] Webserver started at http://127.0.0.1:35029/ using document root <none> and password file <none>
I20260620 14:30:21.876273 21086 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:21.876344 21086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:21.876500 21086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:21.877599 21086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "6344a3e3a15347f5b7d0f73d6b6f75c6"
format_stamp: "Formatted at 2026-06-20 14:30:21 on dist-test-slave-hfc3"
server_key: "aaecec41dd15923756f3cd18dc8a4fdf"
server_key_iv: "30de44914d5d15014c773a9a9351f6be"
server_key_version: "encryptionkey@0"
I20260620 14:30:21.877988 21086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "6344a3e3a15347f5b7d0f73d6b6f75c6"
format_stamp: "Formatted at 2026-06-20 14:30:21 on dist-test-slave-hfc3"
server_key: "aaecec41dd15923756f3cd18dc8a4fdf"
server_key_iv: "30de44914d5d15014c773a9a9351f6be"
server_key_version: "encryptionkey@0"
I20260620 14:30:21.879410 21086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:30:21.880162 21101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:21.880333 21086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:21.880442 21086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "6344a3e3a15347f5b7d0f73d6b6f75c6"
format_stamp: "Formatted at 2026-06-20 14:30:21 on dist-test-slave-hfc3"
server_key: "aaecec41dd15923756f3cd18dc8a4fdf"
server_key_iv: "30de44914d5d15014c773a9a9351f6be"
server_key_version: "encryptionkey@0"
I20260620 14:30:21.880518 21086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:21.906649 21086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:21.908782 21086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:21.908962 21086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:21.913116 21086 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43041
I20260620 14:30:21.913165 21153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43041 every 8 connection(s)
I20260620 14:30:21.913558 21086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:21.914482 21154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:21.916785 21154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6: Bootstrap starting.
I20260620 14:30:21.917383 21154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:21.917673 21154 log.cc:824] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:21.918479 21154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6: No bootstrap required, opened a new log
I20260620 14:30:21.919471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 21086
I20260620 14:30:21.919584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:21.919786 392 external_mini_cluster.cc:1428] Setting key 80c6c66bf73fb81d7cd9e732f6a065f5
I20260620 14:30:21.920665 21154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6344a3e3a15347f5b7d0f73d6b6f75c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43041 } }
I20260620 14:30:21.920837 21154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:21.920877 21154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6344a3e3a15347f5b7d0f73d6b6f75c6, State: Initialized, Role: FOLLOWER
I20260620 14:30:21.921068 21154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6344a3e3a15347f5b7d0f73d6b6f75c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43041 } }
I20260620 14:30:21.921164 21154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:21.921198 21154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:21.921252 21154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:21.922268 21154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6344a3e3a15347f5b7d0f73d6b6f75c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43041 } }
I20260620 14:30:21.922411 21154 leader_election.cc:302] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6344a3e3a15347f5b7d0f73d6b6f75c6; no voters:
I20260620 14:30:21.922641 21154 leader_election.cc:288] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:21.922659 21159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:21.922768 21159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [term 1 LEADER]: Becoming Leader. State: Replica: 6344a3e3a15347f5b7d0f73d6b6f75c6, State: Running, Role: LEADER
I20260620 14:30:21.922909 21154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:21.922940 21159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6344a3e3a15347f5b7d0f73d6b6f75c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43041 } }
I20260620 14:30:21.923384 21159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6344a3e3a15347f5b7d0f73d6b6f75c6. Latest consensus state: current_term: 1 leader_uuid: "6344a3e3a15347f5b7d0f73d6b6f75c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6344a3e3a15347f5b7d0f73d6b6f75c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43041 } } }
I20260620 14:30:21.923503 21159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:21.923516 21160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6344a3e3a15347f5b7d0f73d6b6f75c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6344a3e3a15347f5b7d0f73d6b6f75c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43041 } } }
I20260620 14:30:21.923700 21160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:21.923771 21166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:21.924369 21166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:21.926272 21166 catalog_manager.cc:1347] Generated new cluster ID: 54c8075541d84f378b0d50987dc79597
I20260620 14:30:21.926354 21166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:30:21 dist-test-slave-hfc3 krb5kdc[21073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:30:21.931558 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:21.920329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:53885 (local address 127.0.0.1:43041)
0620 14:30:21.920520 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:21.920529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:21.920563 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:30:21.920658 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:21.920663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:21.920713 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:21.920824 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:21.921696 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:21.922796 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:21.923836 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:21.924464 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:21.924938 (+ 474us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:21.927544 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:21.927548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:21.927548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:21.927561 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:21.929646 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:21.930114 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:21.930118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:21.930118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:21.930264 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:21.930526 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:21.930527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:21.930527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:21.930851 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:21.931024 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:21.931160 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 14:30:21.931293 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
I20260620 14:30:21.932596 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:21.938628 21166 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:21.939338 21166 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:21.941843 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:21.934160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:46955 (local address 127.0.0.1:43041)
0620 14:30:21.934197 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:21.934198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:21.934250 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:30:21.934307 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:21.934307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:21.934317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:21.934415 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:21.935010 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:21.935806 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:21.936784 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:21.937126 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:21.937200 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:21.938798 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:21.938799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:21.938800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:21.938807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:21.940436 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:21.940829 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:21.940830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:21.940830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:21.940877 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:21.941062 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:21.941062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:21.941063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:21.941169 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:21.941262 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:21.941690 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 14:30:21.941752 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:30:21.945221 21166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6344a3e3a15347f5b7d0f73d6b6f75c6: Generated new TSK 0
I20260620 14:30:21.945422 21166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:21.949198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 21086
W20260620 14:30:21.953545 21071 connection.cc:535] client connection to 127.0.0.1:43041 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43041): Connection reset by peer (error 104)
2026-06-20T14:30:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:22 dist-test-slave-hfc3 krb5kdc[21185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:22 dist-test-slave-hfc3 krb5kdc[21185](info): set up 2 sockets
Jun 20 14:30:22 dist-test-slave-hfc3 krb5kdc[21185](info): commencing operation
krb5kdc: starting...
W20260620 14:30:22.986500 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:23 dist-test-slave-hfc3 krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:30:23.112946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:23.196866 21198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:23.197088 21198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:23.197110 21198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:23.198311 21198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:23.198426 21198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:23.198444 21198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:23.198457 21198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:23.198469 21198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:23.199996 21198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:23.200258 21198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:23.200505 21198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:23.205597 21204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:23.205614 21203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:23.205709 21206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:23.205876 21198 server_base.cc:1034] running on GCE node
I20260620 14:30:23.206146 21198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:23.206526 21198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:23.207706 21198 hybrid_clock.cc:648] HybridClock initialized: now 1781965823207696 us; error 32 us; skew 500 ppm
Jun 20 14:30:23 dist-test-slave-hfc3 krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:23.210296 21198 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:30:23.210871 21198 webserver.cc:460] Webserver started at http://127.0.0.1:41521/ using document root <none> and password file <none>
I20260620 14:30:23.211118 21198 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:23.211190 21198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:23.211349 21198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:23.212476 21198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "d97afa3e8cce410090488b25c539b592"
format_stamp: "Formatted at 2026-06-20 14:30:23 on dist-test-slave-hfc3"
server_key: "085d3e50115e0dd3c8802cb8c9bccfea"
server_key_iv: "1f1d7e801535dd82c3ea279e406d9cae"
server_key_version: "encryptionkey@0"
I20260620 14:30:23.212865 21198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "d97afa3e8cce410090488b25c539b592"
format_stamp: "Formatted at 2026-06-20 14:30:23 on dist-test-slave-hfc3"
server_key: "085d3e50115e0dd3c8802cb8c9bccfea"
server_key_iv: "1f1d7e801535dd82c3ea279e406d9cae"
server_key_version: "encryptionkey@0"
I20260620 14:30:23.214311 21198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:23.215215 21213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:23.215390 21198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:30:23.215493 21198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "d97afa3e8cce410090488b25c539b592"
format_stamp: "Formatted at 2026-06-20 14:30:23 on dist-test-slave-hfc3"
server_key: "085d3e50115e0dd3c8802cb8c9bccfea"
server_key_iv: "1f1d7e801535dd82c3ea279e406d9cae"
server_key_version: "encryptionkey@0"
I20260620 14:30:23.215579 21198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:23.235157 21198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:23.237305 21198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:23.237484 21198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:23.241998 21198 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34985
I20260620 14:30:23.242044 21265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34985 every 8 connection(s)
I20260620 14:30:23.242486 21198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:23.243541 21266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:23.245893 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592: Bootstrap starting.
I20260620 14:30:23.246542 21266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:23.246809 21266 log.cc:824] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:23.247514 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592: No bootstrap required, opened a new log
I20260620 14:30:23.248559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 21198
I20260620 14:30:23.248678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:23.248873 392 external_mini_cluster.cc:1428] Setting key 2277147a3b7427f9e2aa0692e396e5c0
I20260620 14:30:23.249698 21266 raft_consensus.cc:348] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97afa3e8cce410090488b25c539b592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34985 } }
I20260620 14:30:23.249886 21266 raft_consensus.cc:374] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:23.249922 21266 raft_consensus.cc:729] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97afa3e8cce410090488b25c539b592, State: Initialized, Role: FOLLOWER
I20260620 14:30:23.250057 21266 consensus_queue.cc:260] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97afa3e8cce410090488b25c539b592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34985 } }
I20260620 14:30:23.250154 21266 raft_consensus.cc:388] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:23.250247 21266 raft_consensus.cc:482] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:23.250311 21266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:23.251174 21266 raft_consensus.cc:504] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97afa3e8cce410090488b25c539b592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34985 } }
I20260620 14:30:23.251291 21266 leader_election.cc:302] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d97afa3e8cce410090488b25c539b592; no voters:
I20260620 14:30:23.251451 21266 leader_election.cc:288] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:23.251499 21271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:23.251667 21271 raft_consensus.cc:686] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [term 1 LEADER]: Becoming Leader. State: Replica: d97afa3e8cce410090488b25c539b592, State: Running, Role: LEADER
I20260620 14:30:23.251713 21266 sys_catalog.cc:564] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:23.251827 21271 consensus_queue.cc:237] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97afa3e8cce410090488b25c539b592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34985 } }
I20260620 14:30:23.252188 21271 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d97afa3e8cce410090488b25c539b592. Latest consensus state: current_term: 1 leader_uuid: "d97afa3e8cce410090488b25c539b592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97afa3e8cce410090488b25c539b592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34985 } } }
I20260620 14:30:23.252287 21271 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:23.252202 21272 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d97afa3e8cce410090488b25c539b592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97afa3e8cce410090488b25c539b592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34985 } } }
I20260620 14:30:23.252422 21272 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592 [sys.catalog]: This master's current role is: LEADER
W20260620 14:30:23.253613 21287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:23.253706 21287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:23.253787 21279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:23.253939 21279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:23.255630 21279 catalog_manager.cc:1347] Generated new cluster ID: 34849f030aaa41f994bd9942f1c3f2b1
I20260620 14:30:23.255704 21279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:23.258643 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:23.249457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:40119 (local address 127.0.0.1:34985)
0620 14:30:23.249701 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:23.249712 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:23.249747 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:30:23.249862 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:23.249869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:23.249933 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:23.250147 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:23.250446 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:23.252239 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:23.253485 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:23.254083 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:23.254207 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:23.255137 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:23.255141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:23.255141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:23.255347 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:23.255380 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:23.255391 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:23.255494 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:23.255578 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:30:23.255608 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":128,"threads_started":1}
I20260620 14:30:23.260123 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:23.262859 21279 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:23.263576 21279 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:23.265446 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:23.261778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:45143 (local address 127.0.0.1:34985)
0620 14:30:23.261828 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:23.261830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:23.261944 (+ 114us) server_negotiation.cc:407] Connection header received
0620 14:30:23.261992 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:23.261993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:23.262003 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:23.262077 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:23.262216 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:23.262949 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:23.264004 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:23.264342 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:23.264417 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:23.265187 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:23.265188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:23.265189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:23.265194 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:23.265211 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:23.265216 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:23.265280 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:23.265351 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:30:23.265367 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":19}
I20260620 14:30:23.273325 21279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d97afa3e8cce410090488b25c539b592: Generated new TSK 0
I20260620 14:30:23.273543 21279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:23.279116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 21198
W20260620 14:30:23.283800 21183 connection.cc:535] client connection to 127.0.0.1:34985 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34985): Connection reset by peer (error 104)
2026-06-20T14:30:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:23 dist-test-slave-hfc3 krb5kdc[21297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:23 dist-test-slave-hfc3 krb5kdc[21297](info): set up 2 sockets
Jun 20 14:30:23 dist-test-slave-hfc3 krb5kdc[21297](info): commencing operation
krb5kdc: starting...
W20260620 14:30:24.330593 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:24 dist-test-slave-hfc3 krb5kdc[21297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:30:24.453054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:24.535802 21310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:24.536105 21310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:24.536151 21310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:24.537364 21310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:24.537446 21310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:24.537472 21310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:24.537493 21310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:24.537525 21310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:24.539098 21310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40581
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:24.539409 21310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:24.539661 21310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:24.545154 21315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:24.545286 21318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:24.545279 21316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:24.545352 21310 server_base.cc:1034] running on GCE node
I20260620 14:30:24.545706 21310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:24.546023 21310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:24.547201 21310 hybrid_clock.cc:648] HybridClock initialized: now 1781965824547194 us; error 39 us; skew 500 ppm
Jun 20 14:30:24 dist-test-slave-hfc3 krb5kdc[21297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:24.549822 21310 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:30:24.550307 21310 webserver.cc:460] Webserver started at http://127.0.0.1:35799/ using document root <none> and password file <none>
I20260620 14:30:24.550585 21310 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:24.550681 21310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:24.550829 21310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:24.551964 21310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "4b09a14b00b446fe9a9b133c0998f87e"
format_stamp: "Formatted at 2026-06-20 14:30:24 on dist-test-slave-hfc3"
server_key: "80b5c08c8aa57ed6e6e6c5b1ae11ebaa"
server_key_iv: "5f83fe569ce96b49e0ac2b0805f6caaf"
server_key_version: "encryptionkey@0"
I20260620 14:30:24.552347 21310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b09a14b00b446fe9a9b133c0998f87e"
format_stamp: "Formatted at 2026-06-20 14:30:24 on dist-test-slave-hfc3"
server_key: "80b5c08c8aa57ed6e6e6c5b1ae11ebaa"
server_key_iv: "5f83fe569ce96b49e0ac2b0805f6caaf"
server_key_version: "encryptionkey@0"
I20260620 14:30:24.553800 21310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:24.554700 21325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:24.554883 21310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:24.554998 21310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "4b09a14b00b446fe9a9b133c0998f87e"
format_stamp: "Formatted at 2026-06-20 14:30:24 on dist-test-slave-hfc3"
server_key: "80b5c08c8aa57ed6e6e6c5b1ae11ebaa"
server_key_iv: "5f83fe569ce96b49e0ac2b0805f6caaf"
server_key_version: "encryptionkey@0"
I20260620 14:30:24.555078 21310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:24.558504 21310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:24.560501 21310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:24.560678 21310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:24.564755 21310 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40581
I20260620 14:30:24.564798 21377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40581 every 8 connection(s)
I20260620 14:30:24.565152 21310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:24.566217 21378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:24.568147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 21310
I20260620 14:30:24.568262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:24.568454 392 external_mini_cluster.cc:1428] Setting key aa9feaa6a08f54fcccccef9b843bc180
I20260620 14:30:24.568786 21378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e: Bootstrap starting.
I20260620 14:30:24.569571 21378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:24.569903 21378 log.cc:824] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:24.571049 21378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e: No bootstrap required, opened a new log
I20260620 14:30:24.571277 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:24.569054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:51653 (local address 127.0.0.1:40581)
0620 14:30:24.569266 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:24.569278 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:24.569313 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:30:24.569972 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:24.569981 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:24.570017 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:24.570194 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:24.570837 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:24.570841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:24.570842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:24.570846 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:24.570893 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:24.570907 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:24.571000 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:24.571031 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 14:30:24.571084 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":118,"threads_started":1}
I20260620 14:30:24.572335 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:24.573499 21378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b09a14b00b446fe9a9b133c0998f87e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40581 } }
I20260620 14:30:24.573705 21378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:24.573758 21378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b09a14b00b446fe9a9b133c0998f87e, State: Initialized, Role: FOLLOWER
I20260620 14:30:24.573894 21378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b09a14b00b446fe9a9b133c0998f87e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40581 } }
I20260620 14:30:24.574005 21378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:24.574041 21378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:24.574095 21378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:24.574904 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:24.573659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:49875 (local address 127.0.0.1:40581)
0620 14:30:24.573697 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:24.573699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:24.573843 (+ 144us) server_negotiation.cc:407] Connection header received
0620 14:30:24.573905 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:24.573905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:24.573910 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:24.573972 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:24.574658 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:24.574659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:24.574659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:24.574663 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:24.574680 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:24.574684 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:24.574745 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:24.574811 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:30:24.574824 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:30:24.574941 21378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b09a14b00b446fe9a9b133c0998f87e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40581 } }
I20260620 14:30:24.575047 21378 leader_election.cc:302] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b09a14b00b446fe9a9b133c0998f87e; no voters:
I20260620 14:30:24.575217 21378 leader_election.cc:288] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:24.575249 21388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:24.575407 21388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [term 1 LEADER]: Becoming Leader. State: Replica: 4b09a14b00b446fe9a9b133c0998f87e, State: Running, Role: LEADER
I20260620 14:30:24.575464 21378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:24.575533 21388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b09a14b00b446fe9a9b133c0998f87e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40581 } }
I20260620 14:30:24.575868 21388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b09a14b00b446fe9a9b133c0998f87e. Latest consensus state: current_term: 1 leader_uuid: "4b09a14b00b446fe9a9b133c0998f87e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b09a14b00b446fe9a9b133c0998f87e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40581 } } }
I20260620 14:30:24.575913 21389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b09a14b00b446fe9a9b133c0998f87e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b09a14b00b446fe9a9b133c0998f87e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40581 } } }
I20260620 14:30:24.575992 21388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:24.576108 21389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:24.576524 21397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:24.577031 21397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:24.578534 21397 catalog_manager.cc:1347] Generated new cluster ID: 2fb4f74d31fa46d183b9d45fd2c8a210
I20260620 14:30:24.578590 21397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:24.590559 21397 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:24.590744 21397 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:24.602154 21397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b09a14b00b446fe9a9b133c0998f87e: Generated new TSK 0
I20260620 14:30:24.602422 21397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:24.606566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 21310
2026-06-20T14:30:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:24 dist-test-slave-hfc3 krb5kdc[21409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:24 dist-test-slave-hfc3 krb5kdc[21409](info): set up 2 sockets
Jun 20 14:30:24 dist-test-slave-hfc3 krb5kdc[21409](info): commencing operation
krb5kdc: starting...
W20260620 14:30:25.650595 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:25 dist-test-slave-hfc3 krb5kdc[21409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:25.773612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36967
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:25.856110 21422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:25.856338 21422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:25.856361 21422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:25.857473 21422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:25.857546 21422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:25.857569 21422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:25.857590 21422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:25.857640 21422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:25.859215 21422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:25.859529 21422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:25.859774 21422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:25.864966 21430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:25.864912 21427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:25.865072 21422 server_base.cc:1034] running on GCE node
W20260620 14:30:25.864912 21428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:25.865608 21422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:25.865922 21422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:25.867101 21422 hybrid_clock.cc:648] HybridClock initialized: now 1781965825867085 us; error 28 us; skew 500 ppm
Jun 20 14:30:25 dist-test-slave-hfc3 krb5kdc[21409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:25.869712 21422 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:25.870237 21422 webserver.cc:460] Webserver started at http://127.0.98.62:33473/ using document root <none> and password file <none>
I20260620 14:30:25.870513 21422 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:25.870591 21422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:25.870754 21422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:25.871910 21422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "571b1d502fc843eda0f1ad4bf89ff303"
format_stamp: "Formatted at 2026-06-20 14:30:25 on dist-test-slave-hfc3"
server_key: "0bc5103623254b01616144e070a727f5"
server_key_iv: "d4805e0fac40e16826df13683bb29e2f"
server_key_version: "encryptionkey@0"
I20260620 14:30:25.872296 21422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "571b1d502fc843eda0f1ad4bf89ff303"
format_stamp: "Formatted at 2026-06-20 14:30:25 on dist-test-slave-hfc3"
server_key: "0bc5103623254b01616144e070a727f5"
server_key_iv: "d4805e0fac40e16826df13683bb29e2f"
server_key_version: "encryptionkey@0"
I20260620 14:30:25.873749 21422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:25.874648 21437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:25.874835 21422 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:25.874938 21422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "571b1d502fc843eda0f1ad4bf89ff303"
format_stamp: "Formatted at 2026-06-20 14:30:25 on dist-test-slave-hfc3"
server_key: "0bc5103623254b01616144e070a727f5"
server_key_iv: "d4805e0fac40e16826df13683bb29e2f"
server_key_version: "encryptionkey@0"
I20260620 14:30:25.875021 21422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:25.889991 21422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:25.892459 21422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:25.892640 21422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:25.896950 21422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36967
I20260620 14:30:25.897001 21489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36967 every 8 connection(s)
I20260620 14:30:25.897380 21422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:25.898291 21490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:25.899562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 21422
I20260620 14:30:25.899659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:25.899859 392 external_mini_cluster.cc:1428] Setting key 21ef3a1c090f612b4b4b6eca5a8d0ddf
I20260620 14:30:25.901077 21490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303: Bootstrap starting.
I20260620 14:30:25.901870 21490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:25.902177 21490 log.cc:824] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:25.902992 21490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303: No bootstrap required, opened a new log
I20260620 14:30:25.905387 21490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b1d502fc843eda0f1ad4bf89ff303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36967 } }
I20260620 14:30:25.905592 21490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:25.905634 21490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571b1d502fc843eda0f1ad4bf89ff303, State: Initialized, Role: FOLLOWER
I20260620 14:30:25.905771 21490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b1d502fc843eda0f1ad4bf89ff303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36967 } }
I20260620 14:30:25.905866 21490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:25.905905 21490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:25.905966 21490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:25.906769 21490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b1d502fc843eda0f1ad4bf89ff303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36967 } }
I20260620 14:30:25.906863 21490 leader_election.cc:302] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 571b1d502fc843eda0f1ad4bf89ff303; no voters:
I20260620 14:30:25.907107 21490 leader_election.cc:288] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:25.907132 21495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [term 1 FOLLOWER]: Leader election won for term 1
Jun 20 14:30:25 dist-test-slave-hfc3 krb5kdc[21409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:25.907251 21495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [term 1 LEADER]: Becoming Leader. State: Replica: 571b1d502fc843eda0f1ad4bf89ff303, State: Running, Role: LEADER
I20260620 14:30:25.907388 21495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b1d502fc843eda0f1ad4bf89ff303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36967 } }
I20260620 14:30:25.907390 21490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:25.907902 21496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "571b1d502fc843eda0f1ad4bf89ff303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b1d502fc843eda0f1ad4bf89ff303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36967 } } }
I20260620 14:30:25.908164 21496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:25.908083 21497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 571b1d502fc843eda0f1ad4bf89ff303. Latest consensus state: current_term: 1 leader_uuid: "571b1d502fc843eda0f1ad4bf89ff303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b1d502fc843eda0f1ad4bf89ff303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36967 } } }
I20260620 14:30:25.908430 21497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:25.908604 21504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:25.909149 21504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:25.910928 21504 catalog_manager.cc:1347] Generated new cluster ID: 0a4251bb03a6438592f043c7342702e5
I20260620 14:30:25.910984 21504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:25.911944 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:25.900372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53161 (local address 127.0.98.62:36967)
0620 14:30:25.900549 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:25.900557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:25.900581 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:30:25.900983 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:25.900989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:25.901034 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:25.901185 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:25.902085 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:25.903274 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:25.904374 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:25.904769 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:25.904853 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:25.907751 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:25.907755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:25.907755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:25.907768 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:25.909959 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:25.910455 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:25.910457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:25.910458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:25.910630 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:25.910951 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:25.910952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:25.910952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:25.911272 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:25.911449 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:25.911608 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:30:25.911729 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260620 14:30:25.913057 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:25.918309 21504 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:25.919080 21504 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:25.924269 21504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 571b1d502fc843eda0f1ad4bf89ff303: Generated new TSK 0
I20260620 14:30:25.924444 21504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:25.925081 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:25.914584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42009 (local address 127.0.98.62:36967)
0620 14:30:25.914627 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:25.914629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:25.917928 (+ 3299us) server_negotiation.cc:407] Connection header received
0620 14:30:25.917981 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:25.917982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:25.917995 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:25.918057 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:25.918791 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:25.919488 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:25.920275 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:25.920511 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:25.920594 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:25.921960 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:25.921962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:25.921962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:25.921970 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:25.923628 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:25.924106 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:25.924107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:25.924107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:25.924175 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:25.924463 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:25.924464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:25.924464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:25.924610 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:25.924678 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:25.924900 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 14:30:25.924980 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":21}
I20260620 14:30:25.927201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 21422
W20260620 14:30:25.931805 21407 connection.cc:535] client connection to 127.0.98.62:36967 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36967): Connection reset by peer (error 104)
2026-06-20T14:30:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:25 dist-test-slave-hfc3 krb5kdc[21521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:25 dist-test-slave-hfc3 krb5kdc[21521](info): set up 2 sockets
Jun 20 14:30:25 dist-test-slave-hfc3 krb5kdc[21521](info): commencing operation
krb5kdc: starting...
W20260620 14:30:27.062726 392 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:27 dist-test-slave-hfc3 krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:27.185868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38099
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:27.267341 21534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:27.267572 21534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:27.267596 21534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:27.268769 21534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:27.268812 21534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:27.268826 21534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:27.268868 21534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:27.268898 21534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:27.270368 21534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38099
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:27.270629 21534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:27.270864 21534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:27.275863 21539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:27.275887 21540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:27.275862 21542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:27.276036 21534 server_base.cc:1034] running on GCE node
I20260620 14:30:27.276317 21534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:27.276609 21534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:27.277792 21534 hybrid_clock.cc:648] HybridClock initialized: now 1781965827277758 us; error 52 us; skew 500 ppm
Jun 20 14:30:27 dist-test-slave-hfc3 krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:27.284982 21534 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:27.285391 21534 webserver.cc:460] Webserver started at http://127.0.98.62:40033/ using document root <none> and password file <none>
I20260620 14:30:27.285630 21534 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:27.285679 21534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:27.285809 21534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:27.286949 21534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "57fe4842f7784ab390e9d05a6e7cfc4c"
format_stamp: "Formatted at 2026-06-20 14:30:27 on dist-test-slave-hfc3"
server_key: "05a71cd15c0dc5143008b58088409872"
server_key_iv: "ef7e1313e2468f836bef0eab38e6eb35"
server_key_version: "encryptionkey@0"
I20260620 14:30:27.287309 21534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "57fe4842f7784ab390e9d05a6e7cfc4c"
format_stamp: "Formatted at 2026-06-20 14:30:27 on dist-test-slave-hfc3"
server_key: "05a71cd15c0dc5143008b58088409872"
server_key_iv: "ef7e1313e2468f836bef0eab38e6eb35"
server_key_version: "encryptionkey@0"
I20260620 14:30:27.288672 21534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:30:27.289480 21549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:27.289674 21534 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:27.289777 21534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "57fe4842f7784ab390e9d05a6e7cfc4c"
format_stamp: "Formatted at 2026-06-20 14:30:27 on dist-test-slave-hfc3"
server_key: "05a71cd15c0dc5143008b58088409872"
server_key_iv: "ef7e1313e2468f836bef0eab38e6eb35"
server_key_version: "encryptionkey@0"
I20260620 14:30:27.289861 21534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:27.299104 21534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:27.301445 21534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:27.301615 21534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:27.305610 21534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38099
I20260620 14:30:27.305670 21601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38099 every 8 connection(s)
I20260620 14:30:27.306026 21534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:27.306941 21602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:27.309259 21602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c: Bootstrap starting.
I20260620 14:30:27.309854 21602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:27.310146 21602 log.cc:824] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:27.310828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 21534
I20260620 14:30:27.310953 21602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c: No bootstrap required, opened a new log
I20260620 14:30:27.310981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:27.311173 392 external_mini_cluster.cc:1428] Setting key 2f8d36fb7627ef3e1a229faaa26ab258
I20260620 14:30:27.312846 21602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57fe4842f7784ab390e9d05a6e7cfc4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } }
I20260620 14:30:27.313032 21602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:27.313073 21602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57fe4842f7784ab390e9d05a6e7cfc4c, State: Initialized, Role: FOLLOWER
I20260620 14:30:27.313265 21602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57fe4842f7784ab390e9d05a6e7cfc4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } }
I20260620 14:30:27.313380 21602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:27.313421 21602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:27.313476 21602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:27.314025 21602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57fe4842f7784ab390e9d05a6e7cfc4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } }
I20260620 14:30:27.314142 21602 leader_election.cc:302] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57fe4842f7784ab390e9d05a6e7cfc4c; no voters:
I20260620 14:30:27.314322 21602 leader_election.cc:288] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:27.314376 21607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:27.314531 21607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [term 1 LEADER]: Becoming Leader. State: Replica: 57fe4842f7784ab390e9d05a6e7cfc4c, State: Running, Role: LEADER
I20260620 14:30:27.314626 21602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:27.314700 21607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57fe4842f7784ab390e9d05a6e7cfc4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } }
I20260620 14:30:27.315028 21607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57fe4842f7784ab390e9d05a6e7cfc4c. Latest consensus state: current_term: 1 leader_uuid: "57fe4842f7784ab390e9d05a6e7cfc4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57fe4842f7784ab390e9d05a6e7cfc4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } } }
I20260620 14:30:27.315263 21607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:27.315183 21608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57fe4842f7784ab390e9d05a6e7cfc4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57fe4842f7784ab390e9d05a6e7cfc4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } } }
I20260620 14:30:27.315688 21608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:27.315686 21616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:27.316295 21616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:27.318025 21616 catalog_manager.cc:1347] Generated new cluster ID: df1947fb9a7d443d8e396c61b88d0225
I20260620 14:30:27.318076 21616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:27.318327 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:27.311747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46037 (local address 127.0.98.62:38099)
0620 14:30:27.311941 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:27.311950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:27.311974 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:30:27.312831 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:27.312838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:27.312887 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:27.313017 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:27.313304 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:27.314835 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:27.316011 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:27.316630 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:27.316728 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:27.317523 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:27.317527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:27.317527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:27.317728 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:27.317765 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:27.317781 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:27.317867 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:27.318129 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 14:30:27.318157 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260620 14:30:27.319293 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:27.323984 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:27.320788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46097 (local address 127.0.98.62:38099)
0620 14:30:27.320826 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:27.320829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:27.320993 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:30:27.321068 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:27.321068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:27.321079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:27.321124 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:27.321234 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:27.321706 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:27.322512 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:27.322887 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:27.322981 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:27.323721 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:27.323722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:27.323722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:27.323726 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:27.323741 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:27.323745 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:27.323800 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:27.323874 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:30:27.323888 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:30:27.326524 21616 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:27.327219 21616 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:27.333519 21616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57fe4842f7784ab390e9d05a6e7cfc4c: Generated new TSK 0
I20260620 14:30:27.333699 21616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:27.342020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 21534
W20260620 14:30:27.346714 21519 connection.cc:535] client connection to 127.0.98.62:38099 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38099): Connection reset by peer (error 104)
2026-06-20T14:30:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:27 dist-test-slave-hfc3 krb5kdc[21633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:27 dist-test-slave-hfc3 krb5kdc[21633](info): set up 2 sockets
Jun 20 14:30:27 dist-test-slave-hfc3 krb5kdc[21633](info): commencing operation
krb5kdc: starting...
W20260620 14:30:28.382542 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:28 dist-test-slave-hfc3 krb5kdc[21633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:28.502797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43561
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:28.585139 21646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:28.585347 21646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:28.585371 21646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:28.587036 21646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:28.587082 21646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:28.587097 21646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:28.587110 21646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:28.587122 21646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:28.588551 21646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:28.588745 21646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:28.588905 21646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:28.593674 21651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:28.593724 21652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:28.593736 21654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:28.593860 21646 server_base.cc:1034] running on GCE node
I20260620 14:30:28.594133 21646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:28.594472 21646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:28.595666 21646 hybrid_clock.cc:648] HybridClock initialized: now 1781965828595654 us; error 31 us; skew 500 ppm
Jun 20 14:30:28 dist-test-slave-hfc3 krb5kdc[21633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:28.598420 21646 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:28.598901 21646 webserver.cc:460] Webserver started at http://127.0.98.62:44223/ using document root <none> and password file <none>
I20260620 14:30:28.599152 21646 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:28.599220 21646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:28.599370 21646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:28.600467 21646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "1e084295ef274c568ef04a2b8f9036f2"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-hfc3"
server_key: "108517f443bac5be6cb74b170322d64c"
server_key_iv: "c8ecd94d5c05558e96df1db4eda0fdf0"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.600835 21646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e084295ef274c568ef04a2b8f9036f2"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-hfc3"
server_key: "108517f443bac5be6cb74b170322d64c"
server_key_iv: "c8ecd94d5c05558e96df1db4eda0fdf0"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.602253 21646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:28.603099 21661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:28.603271 21646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:28.603376 21646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "1e084295ef274c568ef04a2b8f9036f2"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-hfc3"
server_key: "108517f443bac5be6cb74b170322d64c"
server_key_iv: "c8ecd94d5c05558e96df1db4eda0fdf0"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.603456 21646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:28.606897 21646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:28.609193 21646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:28.609345 21646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:28.613701 21646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43561
I20260620 14:30:28.613737 21713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43561 every 8 connection(s)
I20260620 14:30:28.614096 21646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:28.615124 21714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:28.617352 21714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2: Bootstrap starting.
I20260620 14:30:28.617866 21714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:28.618109 21714 log.cc:824] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:28.618530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 21646
I20260620 14:30:28.618650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:28.618822 392 external_mini_cluster.cc:1428] Setting key 3aaf3dde6990ef94469d613d2908fc66
I20260620 14:30:28.618920 21714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2: No bootstrap required, opened a new log
I20260620 14:30:28.621517 21714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e084295ef274c568ef04a2b8f9036f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260620 14:30:28.621697 21714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:28.621747 21714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e084295ef274c568ef04a2b8f9036f2, State: Initialized, Role: FOLLOWER
I20260620 14:30:28.621876 21714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e084295ef274c568ef04a2b8f9036f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260620 14:30:28.622004 21714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:28.622079 21714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:28.622135 21714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:28.622639 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.619392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39207 (local address 127.0.98.62:43561)
0620 14:30:28.620400 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:28.620411 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:28.620452 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:30:28.621041 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:28.621051 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:28.621087 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:28.621258 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:28.621888 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:28.621892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:28.621892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:28.621897 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:28.621949 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:28.621962 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:28.622040 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:28.622110 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:30:28.622154 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":83,"threads_started":1}
I20260620 14:30:28.622813 21714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e084295ef274c568ef04a2b8f9036f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260620 14:30:28.622929 21714 leader_election.cc:302] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e084295ef274c568ef04a2b8f9036f2; no voters:
I20260620 14:30:28.623138 21714 leader_election.cc:288] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:28.623169 21719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:28.623369 21719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [term 1 LEADER]: Becoming Leader. State: Replica: 1e084295ef274c568ef04a2b8f9036f2, State: Running, Role: LEADER
I20260620 14:30:28.623507 21719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e084295ef274c568ef04a2b8f9036f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260620 14:30:28.623522 21714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:28.623620 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:28.624859 21721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e084295ef274c568ef04a2b8f9036f2. Latest consensus state: current_term: 1 leader_uuid: "1e084295ef274c568ef04a2b8f9036f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e084295ef274c568ef04a2b8f9036f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } } }
I20260620 14:30:28.624971 21721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:28.625200 21720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e084295ef274c568ef04a2b8f9036f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e084295ef274c568ef04a2b8f9036f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } } }
I20260620 14:30:28.625276 21720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2 [sys.catalog]: This master's current role is: LEADER
W20260620 14:30:28.625486 21738 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:28.625543 21738 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:28.625603 21739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:28.625773 21739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:28.627475 21739 catalog_manager.cc:1347] Generated new cluster ID: a514af453d8e4930876f946c0142f4d0
I20260620 14:30:28.627571 21739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:28.627746 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.625637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60265 (local address 127.0.98.62:43561)
0620 14:30:28.625682 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:28.625684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:28.626639 (+ 955us) server_negotiation.cc:407] Connection header received
0620 14:30:28.626704 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:28.626704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:28.626710 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:28.626768 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:28.627441 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:28.627443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:28.627443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:28.627447 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:28.627463 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:28.627467 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:28.627519 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:28.627671 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 14:30:28.627684 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:30:28.638597 21739 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:28.638793 21739 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:28.648754 21739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e084295ef274c568ef04a2b8f9036f2: Generated new TSK 0
I20260620 14:30:28.649051 21739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:28.653730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 21646
2026-06-20T14:30:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:28 dist-test-slave-hfc3 krb5kdc[21745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:28 dist-test-slave-hfc3 krb5kdc[21745](info): set up 2 sockets
Jun 20 14:30:28 dist-test-slave-hfc3 krb5kdc[21745](info): commencing operation
krb5kdc: starting...
W20260620 14:30:29.698546 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:29 dist-test-slave-hfc3 krb5kdc[21745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:29.815260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:29.896710 21758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:29.896955 21758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:29.896991 21758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:29.898109 21758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:29.898173 21758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:29.898196 21758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:29.898218 21758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:29.898255 21758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:29.899708 21758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:29.899968 21758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:29.900189 21758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:29.905098 21764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:29.905081 21763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:29.905153 21766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:29.905243 21758 server_base.cc:1034] running on GCE node
I20260620 14:30:29.905592 21758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:29.905829 21758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:29.906963 21758 hybrid_clock.cc:648] HybridClock initialized: now 1781965829906951 us; error 30 us; skew 500 ppm
Jun 20 14:30:29 dist-test-slave-hfc3 krb5kdc[21745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:29.909317 21758 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:29.909819 21758 webserver.cc:460] Webserver started at http://127.0.98.62:37981/ using document root <none> and password file <none>
I20260620 14:30:29.910041 21758 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:29.910106 21758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:29.910247 21758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:29.911329 21758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "1c9dc3fa275a4b76b6cd09fbb0e4e9c1"
format_stamp: "Formatted at 2026-06-20 14:30:29 on dist-test-slave-hfc3"
server_key: "5d000d4cf24ac74813977ffa8f5c43be"
server_key_iv: "8ffc51a8458d5f875ddb57122c513f5d"
server_key_version: "encryptionkey@0"
I20260620 14:30:29.911685 21758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c9dc3fa275a4b76b6cd09fbb0e4e9c1"
format_stamp: "Formatted at 2026-06-20 14:30:29 on dist-test-slave-hfc3"
server_key: "5d000d4cf24ac74813977ffa8f5c43be"
server_key_iv: "8ffc51a8458d5f875ddb57122c513f5d"
server_key_version: "encryptionkey@0"
I20260620 14:30:29.913005 21758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:30:29.913733 21773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:29.913894 21758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:29.913991 21758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "1c9dc3fa275a4b76b6cd09fbb0e4e9c1"
format_stamp: "Formatted at 2026-06-20 14:30:29 on dist-test-slave-hfc3"
server_key: "5d000d4cf24ac74813977ffa8f5c43be"
server_key_iv: "8ffc51a8458d5f875ddb57122c513f5d"
server_key_version: "encryptionkey@0"
I20260620 14:30:29.914074 21758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:29.925504 21758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:29.927337 21758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:29.927493 21758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:29.931622 21758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44883
I20260620 14:30:29.931670 21825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44883 every 8 connection(s)
I20260620 14:30:29.932020 21758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:29.932845 21826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:29.935274 21826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1: Bootstrap starting.
I20260620 14:30:29.935825 21826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:29.936069 21826 log.cc:824] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:29.936739 21826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1: No bootstrap required, opened a new log
I20260620 14:30:29.938316 21826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c9dc3fa275a4b76b6cd09fbb0e4e9c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44883 } }
I20260620 14:30:29.938486 21826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:29.938509 21826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9dc3fa275a4b76b6cd09fbb0e4e9c1, State: Initialized, Role: FOLLOWER
I20260620 14:30:29.938611 21826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c9dc3fa275a4b76b6cd09fbb0e4e9c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44883 } }
I20260620 14:30:29.938661 21826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:29.938681 21826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:29.938714 21826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:29.939221 21826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c9dc3fa275a4b76b6cd09fbb0e4e9c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44883 } }
I20260620 14:30:29.939306 21826 leader_election.cc:302] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c9dc3fa275a4b76b6cd09fbb0e4e9c1; no voters:
I20260620 14:30:29.939431 21826 leader_election.cc:288] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:29.939500 21829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:29.939685 21829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1c9dc3fa275a4b76b6cd09fbb0e4e9c1, State: Running, Role: LEADER
I20260620 14:30:29.939700 21826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:29.939774 21829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c9dc3fa275a4b76b6cd09fbb0e4e9c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44883 } }
I20260620 14:30:29.940272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 21758
I20260620 14:30:29.940287 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c9dc3fa275a4b76b6cd09fbb0e4e9c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c9dc3fa275a4b76b6cd09fbb0e4e9c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44883 } } }
I20260620 14:30:29.940361 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:29.940383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:29.940632 392 external_mini_cluster.cc:1428] Setting key 772a2766d860ed6239bd55d0a5766994
I20260620 14:30:29.940882 21831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c9dc3fa275a4b76b6cd09fbb0e4e9c1. Latest consensus state: current_term: 1 leader_uuid: "1c9dc3fa275a4b76b6cd09fbb0e4e9c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c9dc3fa275a4b76b6cd09fbb0e4e9c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44883 } } }
I20260620 14:30:29.940985 21831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:29.941201 21838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:29.941869 21838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:29.943477 21838 catalog_manager.cc:1347] Generated new cluster ID: ef43404d9d054b0791bda92b6e2675cb
I20260620 14:30:29.943536 21838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:30:29 dist-test-slave-hfc3 krb5kdc[21745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:29.951118 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:29.941171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47639 (local address 127.0.98.62:44883)
0620 14:30:29.941333 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:29.941343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:29.941383 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:30:29.941512 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:29.941519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:29.941581 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:29.941718 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:29.942435 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.943752 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.944515 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.944931 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.945042 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:29.947342 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:29.947347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:29.947347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:29.947359 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:29.949459 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:29.949856 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:29.949859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:29.949859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:29.949959 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:29.950162 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:29.950163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:29.950164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:29.950462 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:29.950638 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:29.950707 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:30:29.950952 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260620 14:30:29.952069 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:29.952775 21838 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:29.953480 21838 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:29.967731 21838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c9dc3fa275a4b76b6cd09fbb0e4e9c1: Generated new TSK 0
I20260620 14:30:29.967957 21838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:29.968952 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:29.953599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39733 (local address 127.0.98.62:44883)
0620 14:30:29.953634 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:29.953636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:29.962558 (+ 8922us) server_negotiation.cc:407] Connection header received
0620 14:30:29.962643 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:29.962644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:29.962656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:29.962718 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:29.963302 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.963835 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.964577 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.964798 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.964870 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:29.966089 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:29.966091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:29.966091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:29.966098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:29.967742 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:29.968171 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:29.968172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:29.968173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:29.968260 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:29.968460 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:29.968461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:29.968461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:29.968595 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:29.968707 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:29.968768 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:30:29.968857 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":17}
I20260620 14:30:29.970839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 21758
2026-06-20T14:30:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:30 dist-test-slave-hfc3 krb5kdc[21858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:30 dist-test-slave-hfc3 krb5kdc[21858](info): set up 2 sockets
Jun 20 14:30:30 dist-test-slave-hfc3 krb5kdc[21858](info): commencing operation
krb5kdc: starting...
W20260620 14:30:30.994549 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:31 dist-test-slave-hfc3 krb5kdc[21858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:31.111219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43221
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:31.194443 21871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:31.194674 21871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:31.194713 21871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:31.195858 21871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:31.195928 21871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:31.195956 21871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:31.195979 21871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:31.196010 21871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:31.197471 21871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:31.197753 21871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:31.197974 21871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:31.202791 21877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:31.202800 21879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:31.202852 21871 server_base.cc:1034] running on GCE node
W20260620 14:30:31.202788 21876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:31.203299 21871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:31.203562 21871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:31.204751 21871 hybrid_clock.cc:648] HybridClock initialized: now 1781965831204741 us; error 34 us; skew 500 ppm
Jun 20 14:30:31 dist-test-slave-hfc3 krb5kdc[21858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:31.207333 21871 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:31.207811 21871 webserver.cc:460] Webserver started at http://127.0.98.62:46361/ using document root <none> and password file <none>
I20260620 14:30:31.208042 21871 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:31.208107 21871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:31.208245 21871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:31.209306 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "3a567330b0384ad49c77b437307246d8"
format_stamp: "Formatted at 2026-06-20 14:30:31 on dist-test-slave-hfc3"
server_key: "a4d3eed1e319aafb9dd8ca539447a7ca"
server_key_iv: "c85e0f5539ef595e9d2804091b9bb95c"
server_key_version: "encryptionkey@0"
I20260620 14:30:31.209682 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "3a567330b0384ad49c77b437307246d8"
format_stamp: "Formatted at 2026-06-20 14:30:31 on dist-test-slave-hfc3"
server_key: "a4d3eed1e319aafb9dd8ca539447a7ca"
server_key_iv: "c85e0f5539ef595e9d2804091b9bb95c"
server_key_version: "encryptionkey@0"
I20260620 14:30:31.211084 21871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:31.211831 21886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:31.211995 21871 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:31.212096 21871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "3a567330b0384ad49c77b437307246d8"
format_stamp: "Formatted at 2026-06-20 14:30:31 on dist-test-slave-hfc3"
server_key: "a4d3eed1e319aafb9dd8ca539447a7ca"
server_key_iv: "c85e0f5539ef595e9d2804091b9bb95c"
server_key_version: "encryptionkey@0"
I20260620 14:30:31.212169 21871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:31.237340 21871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:31.239463 21871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:31.239645 21871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:31.243912 21871 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43221
I20260620 14:30:31.243956 21938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43221 every 8 connection(s)
I20260620 14:30:31.244342 21871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:31.245205 21939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:31.246764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 21871
I20260620 14:30:31.246855 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:31.247045 392 external_mini_cluster.cc:1428] Setting key 8ef9c4fbc93380d1b7f2e079be6d8de0
I20260620 14:30:31.248085 21939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8: Bootstrap starting.
I20260620 14:30:31.248821 21939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:31.249140 21939 log.cc:824] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:31.249876 21939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8: No bootstrap required, opened a new log
I20260620 14:30:31.252100 21939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a567330b0384ad49c77b437307246d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43221 } }
I20260620 14:30:31.252257 21939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:31.252305 21939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a567330b0384ad49c77b437307246d8, State: Initialized, Role: FOLLOWER
I20260620 14:30:31.252477 21939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a567330b0384ad49c77b437307246d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43221 } }
I20260620 14:30:31.252580 21939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:31.252604 21939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:31.252673 21939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:31.253228 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.247623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57627 (local address 127.0.98.62:43221)
0620 14:30:31.247840 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.247850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.247883 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:30:31.248441 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.248452 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.248522 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.248677 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:31.248844 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.250200 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.251027 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.251705 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.251843 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.252501 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:31.252505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:31.252505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:31.252722 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:31.252762 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:31.252780 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:31.252832 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.252949 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.253024 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260620 14:30:31.253345 21939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a567330b0384ad49c77b437307246d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43221 } }
I20260620 14:30:31.253453 21939 leader_election.cc:302] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a567330b0384ad49c77b437307246d8; no voters:
I20260620 14:30:31.253654 21939 leader_election.cc:288] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:31.253687 21944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:31.253796 21944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3a567330b0384ad49c77b437307246d8, State: Running, Role: LEADER
I20260620 14:30:31.253917 21939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:31.253953 21944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a567330b0384ad49c77b437307246d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43221 } }
I20260620 14:30:31.254195 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:31.254403 21945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a567330b0384ad49c77b437307246d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a567330b0384ad49c77b437307246d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43221 } } }
I20260620 14:30:31.254474 21945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:31.254420 21946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a567330b0384ad49c77b437307246d8. Latest consensus state: current_term: 1 leader_uuid: "3a567330b0384ad49c77b437307246d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a567330b0384ad49c77b437307246d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43221 } } }
I20260620 14:30:31.254514 21946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8 [sys.catalog]: This master's current role is: LEADER
W20260620 14:30:31.255774 21960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:31.255861 21960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:31.256120 21953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:31.256372 21953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:31.257735 21953 catalog_manager.cc:1347] Generated new cluster ID: e4e33aeda65545ec8d799012b78c62af
I20260620 14:30:31.257786 21953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:31.259300 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.255617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49797 (local address 127.0.98.62:43221)
0620 14:30:31.255650 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.255651 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.255820 (+ 169us) server_negotiation.cc:407] Connection header received
0620 14:30:31.255913 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.255914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.255924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.255952 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:31.256153 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.256857 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.257854 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.258160 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.258204 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.259028 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:31.259030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:31.259030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:31.259035 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:31.259049 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:31.259053 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:31.259081 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.259206 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.259230 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:30:31.263316 21953 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:31.263821 21953 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:31.267954 21953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a567330b0384ad49c77b437307246d8: Generated new TSK 0
I20260620 14:30:31.268131 21953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:31.270870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 21871
2026-06-20T14:30:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:31 dist-test-slave-hfc3 krb5kdc[21970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:31 dist-test-slave-hfc3 krb5kdc[21970](info): set up 2 sockets
Jun 20 14:30:31 dist-test-slave-hfc3 krb5kdc[21970](info): commencing operation
krb5kdc: starting...
W20260620 14:30:32.310575 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:32 dist-test-slave-hfc3 krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:32.429829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39573
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:32.514039 21983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:32.514248 21983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:32.514273 21983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:32.515460 21983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:32.515511 21983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:32.515527 21983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:32.515540 21983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:32.515552 21983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:32.517006 21983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:32.517205 21983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:32.517375 21983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:32.522267 21988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:32.522274 21983 server_base.cc:1034] running on GCE node
W20260620 14:30:32.522220 21989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:32.522220 21991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:32.522743 21983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:32.523034 21983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:32.524223 21983 hybrid_clock.cc:648] HybridClock initialized: now 1781965832524212 us; error 33 us; skew 500 ppm
Jun 20 14:30:32 dist-test-slave-hfc3 krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:32.532279 21983 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:32.532788 21983 webserver.cc:460] Webserver started at http://127.0.98.62:44763/ using document root <none> and password file <none>
I20260620 14:30:32.533051 21983 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:32.533118 21983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:32.533260 21983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:32.534405 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "dff58626350a4fe196eb6fe787b56ff3"
format_stamp: "Formatted at 2026-06-20 14:30:32 on dist-test-slave-hfc3"
server_key: "9f61eb66fc0491e03754f1273ab8a9cd"
server_key_iv: "dfbdd0ae4e1509c1c09076221ef2a9c2"
server_key_version: "encryptionkey@0"
I20260620 14:30:32.534785 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "dff58626350a4fe196eb6fe787b56ff3"
format_stamp: "Formatted at 2026-06-20 14:30:32 on dist-test-slave-hfc3"
server_key: "9f61eb66fc0491e03754f1273ab8a9cd"
server_key_iv: "dfbdd0ae4e1509c1c09076221ef2a9c2"
server_key_version: "encryptionkey@0"
I20260620 14:30:32.536109 21983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:30:32.536868 21998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:32.537042 21983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:32.537132 21983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "dff58626350a4fe196eb6fe787b56ff3"
format_stamp: "Formatted at 2026-06-20 14:30:32 on dist-test-slave-hfc3"
server_key: "9f61eb66fc0491e03754f1273ab8a9cd"
server_key_iv: "dfbdd0ae4e1509c1c09076221ef2a9c2"
server_key_version: "encryptionkey@0"
I20260620 14:30:32.537206 21983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:32.540634 21983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:32.542716 21983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:32.542874 21983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:32.547068 21983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39573
I20260620 14:30:32.547108 22050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39573 every 8 connection(s)
I20260620 14:30:32.547451 21983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:32.548416 22051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:32.550761 22051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3: Bootstrap starting.
I20260620 14:30:32.551357 22051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:32.551643 22051 log.cc:824] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:32.552347 22051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3: No bootstrap required, opened a new log
I20260620 14:30:32.553928 22051 raft_consensus.cc:348] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff58626350a4fe196eb6fe787b56ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39573 } }
I20260620 14:30:32.554070 22051 raft_consensus.cc:374] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:32.554102 22051 raft_consensus.cc:729] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff58626350a4fe196eb6fe787b56ff3, State: Initialized, Role: FOLLOWER
I20260620 14:30:32.554284 22051 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff58626350a4fe196eb6fe787b56ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39573 } }
I20260620 14:30:32.554409 22051 raft_consensus.cc:388] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:32.554448 22051 raft_consensus.cc:482] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:32.554502 22051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:32.554771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 21983
I20260620 14:30:32.554900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:32.555122 392 external_mini_cluster.cc:1428] Setting key b54bc14cd62ebbca1d7edb0d109283e7
I20260620 14:30:32.555202 22051 raft_consensus.cc:504] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff58626350a4fe196eb6fe787b56ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39573 } }
I20260620 14:30:32.555325 22051 leader_election.cc:302] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff58626350a4fe196eb6fe787b56ff3; no voters:
I20260620 14:30:32.555505 22051 leader_election.cc:288] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:32.555548 22054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:32.556025 22054 raft_consensus.cc:686] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [term 1 LEADER]: Becoming Leader. State: Replica: dff58626350a4fe196eb6fe787b56ff3, State: Running, Role: LEADER
I20260620 14:30:32.556056 22051 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:32.556123 22054 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff58626350a4fe196eb6fe787b56ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39573 } }
I20260620 14:30:32.556519 22054 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff58626350a4fe196eb6fe787b56ff3. Latest consensus state: current_term: 1 leader_uuid: "dff58626350a4fe196eb6fe787b56ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff58626350a4fe196eb6fe787b56ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39573 } } }
I20260620 14:30:32.556650 22054 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:32.556770 22057 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dff58626350a4fe196eb6fe787b56ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff58626350a4fe196eb6fe787b56ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39573 } } }
I20260620 14:30:32.556896 22057 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:32.557018 22061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:32.557499 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:32.555578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41825 (local address 127.0.98.62:39573)
0620 14:30:32.555787 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:32.555796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:32.555835 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:30:32.555962 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:32.555971 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:32.556007 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:32.556430 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:32.556907 (+ 477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:32.556911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:32.556911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:32.556916 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:32.556967 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:32.556983 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:32.557052 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:32.557227 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:30:32.557272 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260620 14:30:32.557706 22061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:32.558743 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:32.560492 22061 catalog_manager.cc:1347] Generated new cluster ID: ec74d57f7956463ba85d77c612e01d8f
I20260620 14:30:32.560541 22061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:32.561496 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:32.560260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56817 (local address 127.0.98.62:39573)
0620 14:30:32.560295 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:32.560297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:32.560468 (+ 171us) server_negotiation.cc:407] Connection header received
0620 14:30:32.560548 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:32.560549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:32.560555 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:32.560612 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:32.561255 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:32.561256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:32.561257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:32.561261 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:32.561275 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:32.561280 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:32.561347 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:32.561399 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:30:32.561414 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:30:32.570240 22061 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:32.570437 22061 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:32.579216 22061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dff58626350a4fe196eb6fe787b56ff3: Generated new TSK 0
I20260620 14:30:32.579408 22061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:32.583751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 21983
2026-06-20T14:30:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1313 ms)
[----------] 18 tests from AuthTokenIssuingTest (23982 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:32 dist-test-slave-hfc3 krb5kdc[22082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:32 dist-test-slave-hfc3 krb5kdc[22082](info): set up 2 sockets
Jun 20 14:30:32 dist-test-slave-hfc3 krb5kdc[22082](info): commencing operation
krb5kdc: starting...
W20260620 14:30:33.615847 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:33 dist-test-slave-hfc3 krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:33.736850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40909
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40909,127.0.98.61:33173,127.0.98.60:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:33.818277 22095 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:30:33.818526 22095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:33.818543 22095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:33.818564 22095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:33.819648 22095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:33.819684 22095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:33.819698 22095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:33.819710 22095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:33.819723 22095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:33.821134 22095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40909,127.0.98.61:33173,127.0.98.60:45983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:33.821318 22095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:33.821512 22095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:33.826520 22101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:33.826592 22103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:33.826520 22100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:33.826545 22095 server_base.cc:1034] running on GCE node
I20260620 14:30:33.826907 22095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:33.827206 22095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:33.828470 22095 hybrid_clock.cc:648] HybridClock initialized: now 1781965833828453 us; error 28 us; skew 500 ppm
Jun 20 14:30:33 dist-test-slave-hfc3 krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:33.831053 22095 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:33.831576 22095 webserver.cc:460] Webserver started at http://127.0.98.62:34227/ using document root <none> and password file <none>
I20260620 14:30:33.831827 22095 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:33.831880 22095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:33.832041 22095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:33.833137 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "1bc17c56eecf4cf2b803ffa20ad6e784"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-hfc3"
server_key: "fc4814127e7dd7d53e0f95298742bce9"
server_key_iv: "9ee3cf8e32bd61da9a4ea79dd5fb530c"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.833515 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "1bc17c56eecf4cf2b803ffa20ad6e784"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-hfc3"
server_key: "fc4814127e7dd7d53e0f95298742bce9"
server_key_iv: "9ee3cf8e32bd61da9a4ea79dd5fb530c"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.835177 22095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:30:33.836021 22110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:33.836342 22095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:33.836462 22095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "1bc17c56eecf4cf2b803ffa20ad6e784"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-hfc3"
server_key: "fc4814127e7dd7d53e0f95298742bce9"
server_key_iv: "9ee3cf8e32bd61da9a4ea79dd5fb530c"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.836544 22095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:33.846073 22095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:33.848158 22095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:33.848331 22095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:33.852905 22095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40909
I20260620 14:30:33.852957 22162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40909 every 8 connection(s)
I20260620 14:30:33.853335 22095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:33.854377 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:33.856024 22163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:33.861600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 22095
I20260620 14:30:33.861720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:33.861936 392 external_mini_cluster.cc:1428] Setting key d6623e385457fdff1425bf03ad6896c3
Jun 20 14:30:33 dist-test-slave-hfc3 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:33.866259 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.856717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40909 (local address 127.0.0.1:32940)
0620 14:30:33.856885 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:33.856929 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:33.857377 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:33.857658 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:33.857660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:33.857922 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:33.858079 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:33.858083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.859255 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.859255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:33.859964 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:33.859966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.860493 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.860494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:33.860548 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:33.861067 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:33.861074 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:33.862879 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:33.864790 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:33.864796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:33.864798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:33.864991 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:33.865222 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:33.865222 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:33.865223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:33.865311 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:33.865670 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:33.865675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:33.865738 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 14:30:33.865841 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:30:33.866071 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":85,"thread_start_us":93,"threads_started":1}
I20260620 14:30:33.866261 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.856789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.0.98.62:40909)
0620 14:30:33.856997 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:33.857001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:33.857033 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:30:33.857523 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:33.857525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:33.857543 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:33.857637 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:33.858173 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.859172 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.860055 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.860399 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.860490 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:33.862977 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:33.862985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:33.862985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:33.862992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:33.864691 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:33.865070 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:33.865073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:33.865074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:33.865155 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:33.865374 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:33.865375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:33.865375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:33.865593 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:33.865679 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:33.865866 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:30:33.866004 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260620 14:30:33.867120 22163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:33.867522 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.867366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33173 (local address 127.0.0.1:47390)
0620 14:30:33.867421 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:33.867480 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:30:33.867682 22113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33173: connect: Connection refused (error 111)
I20260620 14:30:33.868144 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.868022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33173 (local address 127.0.0.1:47398)
0620 14:30:33.868073 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:33.868108 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:30:33.868270 22163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33173: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33173: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:30:33.888648 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33173
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40909,127.0.98.61:33173,127.0.98.60:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:30:33.917541 22163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } attempt: 1
I20260620 14:30:33.918100 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.917933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33173 (local address 127.0.0.1:47402)
0620 14:30:33.918028 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:33.918067 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:30:33.918506 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.918414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33173 (local address 127.0.0.1:47418)
0620 14:30:33.918455 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:33.918482 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 14:30:33.918603 22163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33173: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33173: connect: Connection refused (error 111)
W20260620 14:30:33.971284 22170 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:30:33.971544 22170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:33.971573 22170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:33.971616 22170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:33.972754 22170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:33.972817 22170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:33.972841 22170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:33.972875 22170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:33.972903 22170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:33.974411 22170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40909,127.0.98.61:33173,127.0.98.60:45983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:33.974681 22170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:33.974905 22170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:33.979725 22178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:33.979776 22175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:33.979801 22176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:33.979883 22170 server_base.cc:1034] running on GCE node
I20260620 14:30:33.980247 22170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:33.980576 22170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:33.981738 22170 hybrid_clock.cc:648] HybridClock initialized: now 1781965833981733 us; error 38 us; skew 500 ppm
Jun 20 14:30:33 dist-test-slave-hfc3 krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:33.984311 22170 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:30:33.984752 22170 webserver.cc:460] Webserver started at http://127.0.98.61:44841/ using document root <none> and password file <none>
I20260620 14:30:33.984992 22170 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:33.985057 22170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:33.985198 22170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:33.986232 22170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/data/instance:
uuid: "f4c1eb25d6a64a9886e6b493bc26e613"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-hfc3"
server_key: "3392645c8aa39e91322d8501d2439626"
server_key_iv: "70a17b84a413a7859a00d14774033239"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.986641 22170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/wal/instance:
uuid: "f4c1eb25d6a64a9886e6b493bc26e613"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-hfc3"
server_key: "3392645c8aa39e91322d8501d2439626"
server_key_iv: "70a17b84a413a7859a00d14774033239"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.986753 22163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } attempt: 2
I20260620 14:30:33.987169 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.987013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33173 (local address 127.0.0.1:47424)
0620 14:30:33.987084 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:33.987126 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:30:33.987555 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.987440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33173 (local address 127.0.0.1:47432)
0620 14:30:33.987491 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:33.987521 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 14:30:33.987697 22163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33173: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33173: connect: Connection refused (error 111)
I20260620 14:30:33.988353 22170 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:33.989143 22185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:33.989351 22170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:33.989426 22170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/wal
uuid: "f4c1eb25d6a64a9886e6b493bc26e613"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-hfc3"
server_key: "3392645c8aa39e91322d8501d2439626"
server_key_iv: "70a17b84a413a7859a00d14774033239"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.989526 22170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:34.003082 22170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:34.005153 22170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:34.005299 22170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:34.009397 22170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33173
I20260620 14:30:34.009439 22237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33173 every 8 connection(s)
I20260620 14:30:34.009833 22170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:30:34.010977 22238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:34.012684 22238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:34.013880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 22170
I20260620 14:30:34.014022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-1/wal/instance
I20260620 14:30:34.014226 392 external_mini_cluster.cc:1428] Setting key 19b84e76a089b4bb1807af2bf869bc0c
Jun 20 14:30:34 dist-test-slave-hfc3 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 14:30:34.023689 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.013443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.0.98.62:40909)
0620 14:30:34.013515 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.013517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.013755 (+ 238us) server_negotiation.cc:407] Connection header received
0620 14:30:34.014424 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.014425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.014440 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.014529 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.015275 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.015959 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.017779 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.017970 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.018017 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.020530 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.020532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.020532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.020540 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.022099 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.022610 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.022612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.022612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.022695 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.022960 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.022961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.022962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.023081 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.023157 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.023469 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.023597 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:30:34.023945 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.013422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40909 (local address 127.0.0.1:32950)
0620 14:30:34.013610 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.013658 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.014257 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.014619 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.014623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.014955 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:34.015175 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.015179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.016064 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.016066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.017686 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.017689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.018073 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.018074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.018225 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.018803 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:34.018814 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:34.020433 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:34.022211 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.022219 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.022221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.022521 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.022764 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.022765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.022765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.022879 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.023147 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:34.023154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:34.023300 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.023400 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.023581 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260620 14:30:34.024657 22238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 14:30:34 dist-test-slave-hfc3 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:30:34.032914 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.024949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33173 (local address 127.0.0.1:47446)
0620 14:30:34.024984 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.024998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.025031 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.025377 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.025378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.025524 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:34.025591 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.025592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.026460 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.026462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.027289 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.027292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.027900 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.027901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.027962 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.028550 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:34.028555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:34.029928 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:34.031831 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.031832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.031832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.032001 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.032211 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.032212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.032212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.032266 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.032548 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:34.032549 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:34.032587 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.032661 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.032762 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 14:30:34.032999 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.025050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.0.98.61:33173)
0620 14:30:34.025225 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.025229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.025245 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:30:34.025261 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.025263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.025273 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.025350 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.025665 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.026388 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.027377 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.027817 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.027898 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.030030 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.030037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.030038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.030044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.031717 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.032090 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.032093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.032094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.032147 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.032338 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.032339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.032339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.032481 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.032571 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.032690 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.032832 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260620 14:30:34.033663 22238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:34.034051 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.033883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41770)
0620 14:30:34.033977 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.034024 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 14:30:34.034178 22187 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
I20260620 14:30:34.034662 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.034549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41776)
0620 14:30:34.034594 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.034636 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:30:34.034778 22238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45983: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:30:34.041980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45983
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40909,127.0.98.61:33173,127.0.98.60:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:30:34.078868 22163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } attempt: 3
Jun 20 14:30:34 dist-test-slave-hfc3 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:30:34.083985 22238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } attempt: 1
I20260620 14:30:34.084550 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.084439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41792)
0620 14:30:34.084488 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.084522 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:30:34.084937 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.084844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41804)
0620 14:30:34.084885 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.084911 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:30:34.085101 22238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45983: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
I20260620 14:30:34.086107 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.079296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33173 (local address 127.0.0.1:47456)
0620 14:30:34.079360 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.079374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.079448 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.079595 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.079597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.079770 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:34.079834 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.079837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.080544 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.080545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.081080 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.081082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.081458 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.081459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.081538 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.082043 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:34.082047 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:34.083269 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:34.085073 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.085074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.085074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.085285 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.085485 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.085485 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.085485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.085539 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.085819 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:34.085820 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:34.085880 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.085954 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.086042 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:30:34.086158 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.079391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.0.98.61:33173)
0620 14:30:34.079465 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.079467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.079478 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:30:34.079509 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.079511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.079521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.079590 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.079968 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.080460 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.081167 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.081387 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.081462 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.083358 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.083360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.083360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.083368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.084955 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.085350 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.085351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.085351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.085410 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.085598 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.085599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.085599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.085751 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.085833 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.085993 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.086075 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:30:34.086848 22163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:34.087208 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.087094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41812)
0620 14:30:34.087141 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.087177 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:30:34.087598 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.087508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41822)
0620 14:30:34.087551 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.087579 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:30:34.087749 22163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45983: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
I20260620 14:30:34.118901 22163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } attempt: 1
I20260620 14:30:34.119433 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.119257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41834)
0620 14:30:34.119333 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.119391 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:30:34.119848 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.119740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41838)
0620 14:30:34.119789 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.119823 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 14:30:34.120018 22163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45983: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
W20260620 14:30:34.127673 22245 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:30:34.127875 22245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:34.127892 22245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:34.127918 22245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:34.129001 22245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:34.129035 22245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:34.129050 22245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:34.129062 22245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:34.129074 22245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:34.130558 22245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40909,127.0.98.61:33173,127.0.98.60:45983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:34.130750 22245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:34.130927 22245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:34.135637 22253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:34.135654 22250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:34.135641 22251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:34.136014 22245 server_base.cc:1034] running on GCE node
I20260620 14:30:34.136289 22245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:34.136577 22245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:34.137727 22245 hybrid_clock.cc:648] HybridClock initialized: now 1781965834137709 us; error 35 us; skew 500 ppm
Jun 20 14:30:34 dist-test-slave-hfc3 krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:34.140327 22245 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:30:34.140795 22245 webserver.cc:460] Webserver started at http://127.0.98.60:36513/ using document root <none> and password file <none>
I20260620 14:30:34.141036 22245 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:34.141100 22245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:34.141245 22245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:34.142307 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/data/instance:
uuid: "c76ede11e6e74451870d1ccd4e043973"
format_stamp: "Formatted at 2026-06-20 14:30:34 on dist-test-slave-hfc3"
server_key: "aea84bd6ad937d3276b56163bbaf3885"
server_key_iv: "7e225249cc795f2855b4154e7fa94ba6"
server_key_version: "encryptionkey@0"
I20260620 14:30:34.142725 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/wal/instance:
uuid: "c76ede11e6e74451870d1ccd4e043973"
format_stamp: "Formatted at 2026-06-20 14:30:34 on dist-test-slave-hfc3"
server_key: "aea84bd6ad937d3276b56163bbaf3885"
server_key_iv: "7e225249cc795f2855b4154e7fa94ba6"
server_key_version: "encryptionkey@0"
I20260620 14:30:34.144023 22245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:34.144723 22260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:34.144886 22245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:34.144981 22245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/wal
uuid: "c76ede11e6e74451870d1ccd4e043973"
format_stamp: "Formatted at 2026-06-20 14:30:34 on dist-test-slave-hfc3"
server_key: "aea84bd6ad937d3276b56163bbaf3885"
server_key_iv: "7e225249cc795f2855b4154e7fa94ba6"
server_key_version: "encryptionkey@0"
I20260620 14:30:34.145063 22245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:34.153267 22238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } attempt: 2
I20260620 14:30:34.153712 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.153585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41840)
0620 14:30:34.153644 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.153683 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:30:34.154063 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.153939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41856)
0620 14:30:34.154009 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.154036 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 14:30:34.154147 22238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45983: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45983: connect: Connection refused (error 111)
I20260620 14:30:34.164963 22245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:34.166925 22245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:34.167086 22245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:34.171164 22245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45983
I20260620 14:30:34.171211 22312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45983 every 8 connection(s)
I20260620 14:30:34.171586 22245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:30:34.172551 22313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:34.174140 22313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:34.177163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 22245
I20260620 14:30:34.177244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965454096380-392-0/minicluster-data/master-2/wal/instance
I20260620 14:30:34.177449 392 external_mini_cluster.cc:1428] Setting key 848261fc87b957185c9f4b49918512af
Jun 20 14:30:34 dist-test-slave-hfc3 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:30:34 dist-test-slave-hfc3 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:30:34.183538 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.174907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.0.98.62:40909)
0620 14:30:34.174986 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.174988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.175191 (+ 203us) server_negotiation.cc:407] Connection header received
0620 14:30:34.175670 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.175671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.175680 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.175719 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.176429 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.176887 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.178054 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.178250 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.178356 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.180608 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.180609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.180609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.180616 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.182074 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.182561 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.182562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.182562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.182624 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.182854 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.182856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.182856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.183029 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.183097 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.183336 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.183414 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:30:34.183670 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.174882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40909 (local address 127.0.0.1:32958)
0620 14:30:34.175071 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.175109 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.175576 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.175781 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.175784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.176101 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:34.176346 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.176352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.176964 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.176964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.177935 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.177939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.178389 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.178391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.178502 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.179135 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:34.179144 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:34.180520 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:34.182156 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.182163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.182165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.182492 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.182698 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.182699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.182699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.182793 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.183093 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:34.183098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:34.183208 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.183291 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.183417 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260620 14:30:34.184543 22313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:34.186565 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.177848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.0.98.60:45983)
0620 14:30:34.178040 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.178046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.178071 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:30:34.178101 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.178103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.178123 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.178210 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.178988 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.179690 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.180518 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.180866 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.180918 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.183499 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.183505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.183505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.183511 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.185474 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.185780 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.185783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.185783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.185841 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.186011 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.186011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.186012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.186173 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.186288 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.186434 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.186500 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":122,"threads_started":1}
Jun 20 14:30:34 dist-test-slave-hfc3 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:30:34 dist-test-slave-hfc3 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:34.191777 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.184791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33173 (local address 127.0.0.1:47460)
0620 14:30:34.184907 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.184915 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.184952 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.185130 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.185130 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.185311 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:34.185339 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.185340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.186126 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.186127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.186879 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.186881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.187285 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.187286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.187347 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.187721 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:34.187726 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:34.189117 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:34.190830 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.190831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.190831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.191005 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.191204 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.191205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.191205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.191254 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.191546 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:34.191547 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:34.191584 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.191649 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.191718 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 14:30:34.191972 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.184870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.0.98.61:33173)
0620 14:30:34.184916 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.184918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.184944 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:30:34.185055 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.185056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.185067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.185128 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.185411 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.186056 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.186957 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.187222 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.187270 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.189224 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.189226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.189226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.189233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.190731 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.191076 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.191078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.191078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.191135 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.191316 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.191317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.191317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.191450 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.191535 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.191691 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.191765 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:30:34.192287 22313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:34.195168 22163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } attempt: 2
I20260620 14:30:34.195266 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.187379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.0.98.62:40909)
0620 14:30:34.187413 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.187414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.187422 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:30:34.187480 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.187480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.187490 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.187539 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.188140 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.188870 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.189534 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.189800 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.189891 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.192342 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.192343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.192344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.192351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.194044 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.194390 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.194391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.194391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.194451 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.194669 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.194670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.194670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.194782 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.194927 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.195028 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.195090 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Jun 20 14:30:34 dist-test-slave-hfc3 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:30:34 dist-test-slave-hfc3 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:30:34.200809 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.192552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41870)
0620 14:30:34.192592 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.192606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.192634 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.193053 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.193053 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.193206 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:34.193244 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.193246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.194409 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.194410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.195108 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.195110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.195608 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.195608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.195657 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.196044 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:34.196050 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:34.197328 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:34.199714 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.199714 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.199715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.199876 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.200073 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.200074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.200075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.200150 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.200429 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:34.200430 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:34.200477 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.200559 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.200690 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 14:30:34.200906 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.192794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.0.98.60:45983)
0620 14:30:34.192853 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.192855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.192863 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:30:34.192926 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.192927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.192939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.193038 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.193315 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.194040 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.195193 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.195516 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.195608 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.197417 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.197418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.197418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.197424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.199639 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.199955 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.199956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.199956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.200011 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.200219 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.200220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.200220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.200351 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.200430 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.200609 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.200764 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":33}
Jun 20 14:30:34 dist-test-slave-hfc3 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:30:34.202646 22313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973: Bootstrap starting.
I20260620 14:30:34.203464 22313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:34.203521 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.195416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41878)
0620 14:30:34.195543 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.195554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.195595 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.195881 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.195882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.196059 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:34.196095 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.196097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.196741 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.196742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.197523 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.197526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.197929 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.197930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.197989 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.198444 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:34.198449 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:34.199933 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:34.202354 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.202355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.202356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.202570 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.202831 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.202832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.202833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.202899 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.203177 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:34.203178 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:34.203221 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.203302 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.203401 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 14:30:34.203568 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.195499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.0.98.60:45983)
0620 14:30:34.195765 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.195767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.195774 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:30:34.195794 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.195795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.195805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.195864 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.196172 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.196662 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.197595 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.197864 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.197931 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.200021 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.200022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.200023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.200028 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.202250 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.202641 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.202642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.202642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.202759 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.202978 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.202979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.202979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.203102 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.203171 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.203325 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.203401 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":89,"threads_started":1}
I20260620 14:30:34.203837 22313 log.cc:824] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:34.204725 22313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973: No bootstrap required, opened a new log
I20260620 14:30:34.204732 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.196024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.0.98.61:33173)
0620 14:30:34.196063 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.196065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.196073 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:30:34.196094 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.196095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.196104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.196159 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.196818 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.197516 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.198477 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.198770 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.198816 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.201486 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.201488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.201488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.201495 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.203349 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.203774 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.203775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.203776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.203832 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.204055 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.204056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.204056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.204183 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.204296 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.204550 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.204630 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:30:34.205144 22163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784: Bootstrap starting.
I20260620 14:30:34.205892 22163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:34.206223 22163 log.cc:824] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:34.206447 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:34.206871 22313 raft_consensus.cc:348] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } }
I20260620 14:30:34.207018 22313 raft_consensus.cc:729] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76ede11e6e74451870d1ccd4e043973, State: Initialized, Role: FOLLOWER
I20260620 14:30:34.207062 22163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784: No bootstrap required, opened a new log
I20260620 14:30:34.207108 22313 consensus_queue.cc:260] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } }
I20260620 14:30:34.207409 22313 sys_catalog.cc:564] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:34.207422 22321 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } } }
I20260620 14:30:34.207581 22321 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:34.207470 22137 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32966
I20260620 14:30:34.209053 22163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } }
I20260620 14:30:34.209199 22163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc17c56eecf4cf2b803ffa20ad6e784, State: Initialized, Role: FOLLOWER
I20260620 14:30:34.209290 22163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } }
W20260620 14:30:34.209510 22332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:34.209568 22332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:34.209599 22163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:34.209698 22137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:30:34.209769 22137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:34.210742 22137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } }
I20260620 14:30:34.210994 22137 leader_election.cc:288] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [CANDIDATE]: Term 1 election: Requested vote from peers f4c1eb25d6a64a9886e6b493bc26e613 (127.0.98.61:33173), c76ede11e6e74451870d1ccd4e043973 (127.0.98.60:45983)
W20260620 14:30:34.211426 22345 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:34.211498 22345 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:34.211510 22287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c76ede11e6e74451870d1ccd4e043973"
I20260620 14:30:34.211612 22287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:34.209481 22245 master_runner.cc:186] Error getting master registration for 127.0.98.62:40909: OK, instance_id { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" instance_seqno: 1781965833851596 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:30:34.212237 22245 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:30:34.212245 22287 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc17c56eecf4cf2b803ffa20ad6e784 in term 1.
I20260620 14:30:34.212539 22112 leader_election.cc:302] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc17c56eecf4cf2b803ffa20ad6e784, c76ede11e6e74451870d1ccd4e043973; no voters:
I20260620 14:30:34.212989 22347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } } }
I20260620 14:30:34.213168 22347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:34.213275 22347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:34.213368 22347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [term 1 LEADER]: Becoming Leader. State: Replica: 1bc17c56eecf4cf2b803ffa20ad6e784, State: Running, Role: LEADER
I20260620 14:30:34.213481 22347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } }
I20260620 14:30:34.213950 22334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc17c56eecf4cf2b803ffa20ad6e784. Latest consensus state: current_term: 1 leader_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } } }
I20260620 14:30:34.214035 22334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:34.214214 22354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:34.214480 22354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:34.214802 22095 master_runner.cc:186] Error getting master registration for 127.0.98.61:33173: OK, instance_id { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" instance_seqno: 1781965834008242 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:30:34.217320 22095 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:30:34.217700 22287 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [term 1 FOLLOWER]: Refusing update from remote peer 1bc17c56eecf4cf2b803ffa20ad6e784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:34.217996 22334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:34.219517 22321 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc17c56eecf4cf2b803ffa20ad6e784. Latest consensus state: current_term: 1 leader_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } } }
I20260620 14:30:34.219612 22321 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:34.220460 22334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } } }
I20260620 14:30:34.220548 22334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:34.221029 22354 catalog_manager.cc:1347] Generated new cluster ID: 6b3e1a33368944c08c112197b1d2dbcc
I20260620 14:30:34.221009 22321 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } } }
I20260620 14:30:34.221079 22354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:34.221083 22321 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:34.221448 22356 mvcc.cc:205] Tried to move back new op lower bound from 7298932056953131008 to 7298932056939155456. Current Snapshot: MvccSnapshot[applied={T|T < 7298932056953131008}]
I20260620 14:30:34.221684 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.213544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.0.98.62:40909)
0620 14:30:34.213587 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.213589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.213908 (+ 319us) server_negotiation.cc:407] Connection header received
0620 14:30:34.214408 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.214409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.214420 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.214452 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.215255 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.215722 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.216372 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.216640 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.216709 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.218519 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.218520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.218520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.218527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.220150 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.220562 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.220563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.220563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.220623 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.220849 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.220850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.220850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.221001 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.221128 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.221517 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.221596 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:30:34.233834 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.226590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.0.98.60:45983)
0620 14:30:34.226649 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.226652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.226661 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:30:34.226682 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.226683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.226693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.226744 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.227390 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.228135 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.229089 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.229341 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.229417 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.230932 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.230933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.230933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.230940 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.232635 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.233039 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.233040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.233041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.233111 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.233295 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.233296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.233296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.233432 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.233547 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.233658 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.233733 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:30:34.241508 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.226857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.0.98.61:33173)
0620 14:30:34.226908 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.226910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.233857 (+ 6947us) server_negotiation.cc:407] Connection header received
0620 14:30:34.234305 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.234306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.234318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.234372 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.235055 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.235510 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.236423 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.237005 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.237103 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.238517 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.238518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.238519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.238526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.240141 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.240540 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.240541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.240541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.240598 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.240835 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.240836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.240836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.241009 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.241126 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.241292 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.241371 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:30:34.245272 22238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } attempt: 3
Jun 20 14:30:34 dist-test-slave-hfc3 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:30:34.253523 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.245522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45983 (local address 127.0.0.1:41896)
0620 14:30:34.245572 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.245583 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.245621 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.245749 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.245749 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.245930 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:34.245973 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.245976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.246885 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.246886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.247707 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.247709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.248201 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.248202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.248272 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.248757 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:34.248763 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:34.250423 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:34.252434 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.252435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.252436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.252646 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.252837 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.252838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.252838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.252915 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.253201 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:34.253202 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:34.253243 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.253324 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.253425 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:30:34.253523 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.245582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.0.98.60:45983)
0620 14:30:34.245635 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.245637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.245644 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:30:34.245680 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.245681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.245692 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.245743 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.246046 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.246815 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.247792 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.248142 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.248204 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.250528 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.250530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.250530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.250537 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.252326 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.252725 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.252726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.252726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.252779 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.252991 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.252992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.252992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.253129 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.253186 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.253347 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.253427 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:30:34.255676 22238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613: Bootstrap starting.
I20260620 14:30:34.256153 22354 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:34.256482 22238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:34.256793 22238 log.cc:824] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:34.256872 22354 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:34.262534 22354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784: Generated new TSK 0
I20260620 14:30:34.262801 22354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:34.263113 22238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613: No bootstrap required, opened a new log
I20260620 14:30:34.265234 22238 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } }
I20260620 14:30:34.265379 22238 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c1eb25d6a64a9886e6b493bc26e613, State: Initialized, Role: FOLLOWER
I20260620 14:30:34.265471 22238 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } }
I20260620 14:30:34.265914 22238 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:34.265978 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f4c1eb25d6a64a9886e6b493bc26e613"
I20260620 14:30:34.266158 22212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:34.267077 22212 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc17c56eecf4cf2b803ffa20ad6e784 in term 1.
I20260620 14:30:34.265858 22362 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } } }
I20260620 14:30:34.267396 22362 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:34.267974 22211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [term 1 FOLLOWER]: Refusing update from remote peer 1bc17c56eecf4cf2b803ffa20ad6e784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:34.268247 22334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:30:34.268532 22373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:34.268630 22373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:34.271627 22362 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc17c56eecf4cf2b803ffa20ad6e784. Latest consensus state: current_term: 1 leader_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } } }
I20260620 14:30:34.271728 22362 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:34.271783 22362 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } } }
I20260620 14:30:34.271832 22362 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:34.273039 22376 mvcc.cc:205] Tried to move back new op lower bound from 7298932057135017984 to 7298932056939155456. Current Snapshot: MvccSnapshot[applied={T|T < 7298932056953131008}]
W20260620 14:30:34.292135 22166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.287734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.0.98.62:40909)
0620 14:30:34.287783 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.287786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.291325 (+ 3539us) server_negotiation.cc:407] Connection header received
0620 14:30:34.291461 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.291463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.291476 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.291509 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.292050 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32974: BlockingRecv error: recv got EOF from 127.0.0.1:32974 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 14:30:34.293056 22242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.290655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.0.98.61:33173)
0620 14:30:34.290701 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.290703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.292449 (+ 1746us) server_negotiation.cc:407] Connection header received
0620 14:30:34.292546 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.292547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.292557 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.292586 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.292981 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47496: BlockingRecv error: recv got EOF from 127.0.0.1:47496 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 14:30:34.293809 22318 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.291034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.0.98.60:45983)
0620 14:30:34.291076 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.291078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.293280 (+ 2202us) server_negotiation.cc:407] Connection header received
0620 14:30:34.293378 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.293379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.293389 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.293419 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.293746 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41906: BlockingRecv error: recv got EOF from 127.0.0.1:41906 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:30:34.299520 22242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.298666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.0.98.61:33173)
0620 14:30:34.298716 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.298718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.298941 (+ 223us) server_negotiation.cc:407] Connection header received
0620 14:30:34.299056 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.299057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.299077 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.299107 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.299462 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47498: BlockingRecv error: recv got EOF from 127.0.0.1:47498 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:30:34.307118 22242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.302727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.0.98.61:33173)
0620 14:30:34.302784 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.302785 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.306534 (+ 3749us) server_negotiation.cc:407] Connection header received
0620 14:30:34.306641 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.306642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.306653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.306682 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.307059 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47512: BlockingRecv error: recv got EOF from 127.0.0.1:47512 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 14:30:34.315550 22242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.314671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.0.98.61:33173)
0620 14:30:34.314720 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.314722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.314984 (+ 262us) server_negotiation.cc:407] Connection header received
0620 14:30:34.315085 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.315086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.315097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.315126 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.315494 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47516: BlockingRecv error: recv got EOF from 127.0.0.1:47516 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:30:34.321640 22242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.320805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.0.98.61:33173)
0620 14:30:34.320840 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.320842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.321096 (+ 254us) server_negotiation.cc:407] Connection header received
0620 14:30:34.321195 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.321196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.321206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.321235 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.321584 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47528: BlockingRecv error: recv got EOF from 127.0.0.1:47528 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:30:34.331830 22242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.330918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.0.98.61:33173)
0620 14:30:34.330958 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.330960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.331233 (+ 273us) server_negotiation.cc:407] Connection header received
0620 14:30:34.331348 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.331349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.331360 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.331390 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.331771 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47532: BlockingRecv error: recv got EOF from 127.0.0.1:47532 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:30:34.350469 22242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.349481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.0.98.61:33173)
0620 14:30:34.349526 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.349528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.349832 (+ 304us) server_negotiation.cc:407] Connection header received
0620 14:30:34.349943 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.349945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.349956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.349987 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.350408 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47534: BlockingRecv error: recv got EOF from 127.0.0.1:47534 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:30:34.385008 22242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.384017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.0.98.61:33173)
0620 14:30:34.384066 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.384068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.384378 (+ 310us) server_negotiation.cc:407] Connection header received
0620 14:30:34.384487 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.384488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.384500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.384531 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.384945 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47548: BlockingRecv error: recv got EOF from 127.0.0.1:47548 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 14:30:34.451222 22242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.450499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47560 (local address 127.0.98.61:33173)
0620 14:30:34.450557 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.450559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.450668 (+ 109us) server_negotiation.cc:407] Connection header received
0620 14:30:34.450719 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.450720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.450730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.450786 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.451150 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47560: BlockingRecv error: recv got EOF from 127.0.0.1:47560 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 14:30:34.581197 22242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.580437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.0.98.61:33173)
0620 14:30:34.580516 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.580519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.580637 (+ 118us) server_negotiation.cc:407] Connection header received
0620 14:30:34.580716 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.580717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.580730 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.580800 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.581149 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47568: BlockingRecv error: recv got EOF from 127.0.0.1:47568 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 14:30:34.839540 22242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:34.838614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47572 (local address 127.0.98.61:33173)
0620 14:30:34.838688 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.838691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.838768 (+ 77us) server_negotiation.cc:407] Connection header received
0620 14:30:34.838840 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.838841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.838852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.838895 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.839438 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47572: BlockingRecv error: recv got EOF from 127.0.0.1:47572 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:30:34.885327 22440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bc17c56eecf4cf2b803ffa20ad6e784" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40909 } } peers { permanent_uuid: "f4c1eb25d6a64a9886e6b493bc26e613" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33173 } } peers { permanent_uuid: "c76ede11e6e74451870d1ccd4e043973" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45983 } } }
I20260620 14:30:34.885516 22440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc17c56eecf4cf2b803ffa20ad6e784 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:35.209996 22332 catalog_manager.cc:1259] Loaded cluster ID: 6b3e1a33368944c08c112197b1d2dbcc
I20260620 14:30:35.210075 22332 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973: loading cluster ID for follower catalog manager: success
I20260620 14:30:35.211040 22332 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973: acquiring CA information for follower catalog manager: success
I20260620 14:30:35.211361 22332 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c76ede11e6e74451870d1ccd4e043973: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:30:35.269112 22373 catalog_manager.cc:1259] Loaded cluster ID: 6b3e1a33368944c08c112197b1d2dbcc
I20260620 14:30:35.269196 22373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613: loading cluster ID for follower catalog manager: success
I20260620 14:30:35.270160 22373 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613: acquiring CA information for follower catalog manager: success
I20260620 14:30:35.270509 22373 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f4c1eb25d6a64a9886e6b493bc26e613: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:30:35.355192 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:35.352684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.0.98.61:33173)
0620 14:30:35.352787 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:35.352789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:35.352815 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:30:35.352904 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:35.352904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:35.352956 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:35.353032 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:35.353169 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:35.353662 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:35.354474 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:35.354663 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:35.354715 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:35.354842 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:35.354950 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:35.355007 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:35.355045 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 14:30:35.355061 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":50,"threads_started":1}
I20260620 14:30:35.359758 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:35.357136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.0.98.60:45983)
0620 14:30:35.357271 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:35.357273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:35.357313 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:30:35.357430 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:35.357431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:35.357469 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:35.357557 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:35.357654 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:35.358175 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:35.358984 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:35.359185 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:35.359268 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:35.359426 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:35.359534 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:35.359618 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:35.359641 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 14:30:35.359661 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":54,"threads_started":1}
I20260620 14:30:35.364341 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:35.361464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.0.98.61:33173)
0620 14:30:35.361507 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:35.361509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:35.361605 (+ 96us) server_negotiation.cc:407] Connection header received
0620 14:30:35.361659 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:35.361660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:35.361672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:35.361723 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:35.361856 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:35.362434 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:35.363482 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:35.363798 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:35.363864 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:35.364011 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:35.364049 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:35.364099 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:35.364211 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:30:35.364239 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:30:35.364521 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:35.361455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.0.98.60:45983)
0620 14:30:35.361492 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:35.361495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:35.361639 (+ 144us) server_negotiation.cc:407] Connection header received
0620 14:30:35.361727 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:35.361728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:35.361737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:35.361785 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:35.361906 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:35.362676 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:35.363661 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:35.363957 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:35.364033 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:35.364246 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:35.364280 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:35.364309 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:35.364420 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 14:30:35.364442 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:30:35.365155 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:35.361337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32984 (local address 127.0.98.62:40909)
0620 14:30:35.361470 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:35.361472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:35.361537 (+ 65us) server_negotiation.cc:407] Connection header received
0620 14:30:35.361671 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:35.361672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:35.361724 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:35.361798 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:35.361982 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:35.362769 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:35.363571 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:35.363850 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:35.363923 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:35.364156 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:35.364278 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:35.364337 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:35.364394 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 14:30:35.364419 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260620 14:30:35.366926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 22095
I20260620 14:30:35.372771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 22170
I20260620 14:30:35.377908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 22245
2026-06-20T14:30:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2795 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:35 dist-test-slave-hfc3 krb5kdc[22464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:35 dist-test-slave-hfc3 krb5kdc[22464](info): set up 2 sockets
Jun 20 14:30:35 dist-test-slave-hfc3 krb5kdc[22464](info): commencing operation
krb5kdc: starting...
W20260620 14:30:36.414656 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:36 dist-test-slave-hfc3 krb5kdc[22464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:36.533200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46095
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46095,127.0.98.61:43853,127.0.98.60:37379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:36.617540 22477 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:30:36.617816 22477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:36.617846 22477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:36.617898 22477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:36.619104 22477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:36.619182 22477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:36.619205 22477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:36.619226 22477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:36.619257 22477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:36.620759 22477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46095,127.0.98.61:43853,127.0.98.60:37379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:36.621048 22477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:36.621273 22477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:36.626312 22483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:36.626291 22485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:36.626291 22482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:36.626324 22477 server_base.cc:1034] running on GCE node
I20260620 14:30:36.626853 22477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:36.627168 22477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:36.628341 22477 hybrid_clock.cc:648] HybridClock initialized: now 1781965836628328 us; error 39 us; skew 500 ppm
Jun 20 14:30:36 dist-test-slave-hfc3 krb5kdc[22464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:36.631002 22477 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:36.631491 22477 webserver.cc:460] Webserver started at http://127.0.98.62:34557/ using document root <none> and password file <none>
I20260620 14:30:36.631717 22477 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:36.631764 22477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:36.631917 22477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:36.633013 22477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/data/instance:
uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b"
format_stamp: "Formatted at 2026-06-20 14:30:36 on dist-test-slave-hfc3"
server_key: "bc22fe3f04698d43816f74852528664f"
server_key_iv: "97024f6dc71170ab831ee913e0aaec7d"
server_key_version: "encryptionkey@0"
I20260620 14:30:36.633388 22477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b"
format_stamp: "Formatted at 2026-06-20 14:30:36 on dist-test-slave-hfc3"
server_key: "bc22fe3f04698d43816f74852528664f"
server_key_iv: "97024f6dc71170ab831ee913e0aaec7d"
server_key_version: "encryptionkey@0"
I20260620 14:30:36.634819 22477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:30:36.635583 22492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:36.635757 22477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:30:36.635859 22477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/wal
uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b"
format_stamp: "Formatted at 2026-06-20 14:30:36 on dist-test-slave-hfc3"
server_key: "bc22fe3f04698d43816f74852528664f"
server_key_iv: "97024f6dc71170ab831ee913e0aaec7d"
server_key_version: "encryptionkey@0"
I20260620 14:30:36.635941 22477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:36.653651 22477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:36.655920 22477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:36.656092 22477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:36.660375 22477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46095
I20260620 14:30:36.660426 22544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46095 every 8 connection(s)
I20260620 14:30:36.660807 22477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:36.661808 22545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:36.663434 22545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:36.668438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 22477
I20260620 14:30:36.668589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:36.668845 392 external_mini_cluster.cc:1428] Setting key 9608d4152e43a769ab455eaf0f024c65
Jun 20 14:30:36 dist-test-slave-hfc3 krb5kdc[22464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:36.673484 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.664124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46095 (local address 127.0.0.1:60724)
0620 14:30:36.664285 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.664321 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:36.664783 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:36.665039 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:36.665041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:36.665287 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:36.665441 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:36.665444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.666482 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.666483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:36.667168 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:36.667170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.667646 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.667647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:36.667703 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.668170 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:36.668178 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:36.669887 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:36.672000 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:36.672007 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:36.672009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:36.672196 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:36.672461 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:36.672461 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:36.672462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:36.672542 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:36.672907 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:36.672912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:36.672975 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 14:30:36.673082 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:30:36.673296 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":71,"thread_start_us":86,"threads_started":1}
I20260620 14:30:36.673488 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.664146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.0.98.62:46095)
0620 14:30:36.664355 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:36.664358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:36.664383 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:30:36.664911 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:36.664912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:36.664936 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:36.665015 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:36.665523 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.666394 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.667237 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.667559 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.667637 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.669992 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:36.669998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:36.669999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:36.670012 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:36.671907 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.672304 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.672307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.672307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.672395 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.672621 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.672622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.672622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.672839 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:36.672930 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:36.673110 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:30:36.673239 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260620 14:30:36.674417 22545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:36.674798 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.674659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43853 (local address 127.0.0.1:56648)
0620 14:30:36.674715 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.674768 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:30:36.674918 22496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43853: connect: Connection refused (error 111)
I20260620 14:30:36.675343 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.675251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43853 (local address 127.0.0.1:56658)
0620 14:30:36.675297 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.675323 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:30:36.675467 22545 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43853: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43853: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:30:36.693405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43853
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46095,127.0.98.61:43853,127.0.98.60:37379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:30:36.724713 22545 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } attempt: 1
I20260620 14:30:36.725279 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.725147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43853 (local address 127.0.0.1:56660)
0620 14:30:36.725210 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.725250 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:30:36.725620 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.725526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43853 (local address 127.0.0.1:56664)
0620 14:30:36.725571 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.725601 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:30:36.725708 22545 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43853: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43853: connect: Connection refused (error 111)
W20260620 14:30:36.775719 22552 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:30:36.775980 22552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:36.776010 22552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:36.776051 22552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:36.777171 22552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:36.777236 22552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:36.777261 22552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:36.777292 22552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:36.777320 22552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:36.778949 22552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46095,127.0.98.61:43853,127.0.98.60:37379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:36.779225 22552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:36.779451 22552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:36.784274 22560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:36.784281 22558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:36.784299 22552 server_base.cc:1034] running on GCE node
W20260620 14:30:36.784289 22557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:36.784783 22552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:36.785089 22552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:36.786257 22552 hybrid_clock.cc:648] HybridClock initialized: now 1781965836786237 us; error 35 us; skew 500 ppm
Jun 20 14:30:36 dist-test-slave-hfc3 krb5kdc[22464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:36.788879 22552 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:30:36.789315 22552 webserver.cc:460] Webserver started at http://127.0.98.61:35339/ using document root <none> and password file <none>
I20260620 14:30:36.789507 22552 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:36.789546 22552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:36.789634 22552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:36.790714 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/data/instance:
uuid: "f2baf75ba96f4fa7881edc8f1feb0b56"
format_stamp: "Formatted at 2026-06-20 14:30:36 on dist-test-slave-hfc3"
server_key: "d5ea31b58db8fc2494e6247c48223f8a"
server_key_iv: "b8cf5823b6c03b7aba37c5fb4e8ef843"
server_key_version: "encryptionkey@0"
I20260620 14:30:36.791030 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/wal/instance:
uuid: "f2baf75ba96f4fa7881edc8f1feb0b56"
format_stamp: "Formatted at 2026-06-20 14:30:36 on dist-test-slave-hfc3"
server_key: "d5ea31b58db8fc2494e6247c48223f8a"
server_key_iv: "b8cf5823b6c03b7aba37c5fb4e8ef843"
server_key_version: "encryptionkey@0"
I20260620 14:30:36.792318 22552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:30:36.793056 22567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:36.793226 22552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:30:36.793329 22552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/wal
uuid: "f2baf75ba96f4fa7881edc8f1feb0b56"
format_stamp: "Formatted at 2026-06-20 14:30:36 on dist-test-slave-hfc3"
server_key: "d5ea31b58db8fc2494e6247c48223f8a"
server_key_iv: "b8cf5823b6c03b7aba37c5fb4e8ef843"
server_key_version: "encryptionkey@0"
I20260620 14:30:36.793403 22552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:36.793835 22545 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } attempt: 2
I20260620 14:30:36.794229 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.794081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43853 (local address 127.0.0.1:56678)
0620 14:30:36.794146 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.794192 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:30:36.794610 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.794514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43853 (local address 127.0.0.1:56694)
0620 14:30:36.794558 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.794588 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:30:36.794715 22545 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43853: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43853: connect: Connection refused (error 111)
I20260620 14:30:36.815232 22552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:36.817636 22552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:36.817804 22552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:36.821832 22552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43853
I20260620 14:30:36.821872 22619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43853 every 8 connection(s)
I20260620 14:30:36.822284 22552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:30:36.823285 22620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:36.824777 22620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:36.828946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 22552
I20260620 14:30:36.829071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-1/wal/instance
I20260620 14:30:36.829315 392 external_mini_cluster.cc:1428] Setting key ffc01b9fa792d60ebecc0e56620815a0
Jun 20 14:30:36 dist-test-slave-hfc3 krb5kdc[22464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:36.834093 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.825427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.0.98.62:46095)
0620 14:30:36.825494 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:36.825495 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:36.825727 (+ 232us) server_negotiation.cc:407] Connection header received
0620 14:30:36.826184 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:36.826185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:36.826198 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:36.826262 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:36.826918 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.827432 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.828399 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.828605 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.828657 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.831224 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:36.831226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:36.831226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:36.831234 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:36.832648 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.833100 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.833101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.833102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.833167 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.833427 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.833428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.833428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.833549 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:36.833617 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:36.833902 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:30:36.833993 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:30:36.834316 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.825397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46095 (local address 127.0.0.1:60726)
0620 14:30:36.825568 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.825612 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:36.826082 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:36.826384 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:36.826389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:36.826648 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:36.826840 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:36.826844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.827521 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.827522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:36.828330 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:36.828331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.828686 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.828687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:36.828856 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.829426 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:36.829437 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:36.831101 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:36.832734 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:36.832742 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:36.832744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:36.833018 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:36.833233 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:36.833234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:36.833234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:36.833350 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:36.833620 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:36.833627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:36.833773 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 14:30:36.833882 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:30:36.834060 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260620 14:30:36.835120 22620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 14:30:36 dist-test-slave-hfc3 krb5kdc[22464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:30:36.844255 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.835484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.0.98.61:43853)
0620 14:30:36.835664 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:36.835667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:36.835690 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:30:36.835707 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:36.835708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:36.835719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:36.835781 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:36.836115 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.836803 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.837874 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.838273 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.838369 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.841206 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:36.841217 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:36.841217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:36.841225 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:36.842978 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.843346 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.843350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.843350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.843405 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.843621 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.843622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.843623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.843777 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:36.843867 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:36.843992 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:30:36.844126 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260620 14:30:36.844262 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.835392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43853 (local address 127.0.0.1:56710)
0620 14:30:36.835440 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.835451 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:36.835491 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:36.835811 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:36.835812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:36.835977 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:36.836039 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:36.836041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.836884 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.836886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:36.837783 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:36.837786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.838385 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.838386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:36.838451 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.838934 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:36.838940 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:36.841101 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:36.843085 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:36.843085 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:36.843086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:36.843253 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:36.843501 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:36.843502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:36.843502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:36.843556 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:36.843843 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:36.843844 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:36.843883 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:30:36.843962 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:30:36.844175 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":113}
I20260620 14:30:36.845345 22620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:36.845717 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.845593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34602)
0620 14:30:36.845639 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.845688 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:30:36.845863 22569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
I20260620 14:30:36.846303 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.846209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34604)
0620 14:30:36.846255 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.846280 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:30:36.846428 22620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37379: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:30:36.855482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLb3bhr/build/release/bin/kudu
/tmp/dist-test-taskLb3bhr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37379
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46095,127.0.98.61:43853,127.0.98.60:37379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:30:36.885879 22545 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } attempt: 3
Jun 20 14:30:36 dist-test-slave-hfc3 krb5kdc[22464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:30:36.893410 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.886265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43853 (local address 127.0.0.1:56714)
0620 14:30:36.886354 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.886367 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:36.886451 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:36.886635 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:36.886636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:36.886810 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:36.886881 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:36.886883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.887634 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.887636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:36.888222 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:36.888223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.888641 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.888642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:36.888735 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.889233 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:36.889238 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:36.890567 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:36.892478 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:36.892479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:36.892479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:36.892659 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:36.892863 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:36.892864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:36.892864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:36.892916 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:36.893171 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:36.893171 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:36.893210 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:30:36.893261 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:30:36.893337 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:30:36.893543 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.886366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.0.98.61:43853)
0620 14:30:36.886444 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:36.886446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:36.886460 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:30:36.886528 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:36.886530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:36.886543 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:36.886626 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:36.886975 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.887543 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.888310 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.888560 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.888645 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.890659 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:36.890661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:36.890661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:36.890669 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:36.892379 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.892730 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.892731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.892732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.892795 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.892971 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.892971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.892972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.893110 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:36.893182 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:36.893306 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:30:36.893411 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:30:36.893931 22545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:36.894286 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.894186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34618)
0620 14:30:36.894230 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.894266 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:30:36.894681 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.894599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34624)
0620 14:30:36.894642 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.894666 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 14:30:36.894783 22545 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37379: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
I20260620 14:30:36.895638 22620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } attempt: 1
I20260620 14:30:36.895982 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.895869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34640)
0620 14:30:36.895915 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.895944 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:30:36.896306 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.896219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34650)
0620 14:30:36.896261 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.896285 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:30:36.896445 22620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37379: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
I20260620 14:30:36.925922 22545 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } attempt: 1
I20260620 14:30:36.926479 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.926321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34656)
0620 14:30:36.926409 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.926450 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:30:36.926954 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.926844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34670)
0620 14:30:36.926894 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.926933 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:30:36.927131 22545 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37379: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
W20260620 14:30:36.945082 22627 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:30:36.945333 22627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:36.945353 22627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:36.945408 22627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:36.946779 22627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:36.946833 22627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:36.946851 22627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:36.946883 22627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:36.946913 22627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:36.948593 22627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46095,127.0.98.61:43853,127.0.98.60:37379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:30:36.948891 22627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:36.949157 22627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:30:36.954631 22627 server_base.cc:1034] running on GCE node
W20260620 14:30:36.954612 22633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:36.954607 22635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:36.954612 22632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:36.955112 22627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:36.955410 22627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:36.956584 22627 hybrid_clock.cc:648] HybridClock initialized: now 1781965836956566 us; error 36 us; skew 500 ppm
Jun 20 14:30:36 dist-test-slave-hfc3 krb5kdc[22464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:36.959100 22627 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:30:36.959547 22627 webserver.cc:460] Webserver started at http://127.0.98.60:38897/ using document root <none> and password file <none>
I20260620 14:30:36.959770 22627 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:36.959817 22627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:36.959983 22627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:36.961068 22627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/data/instance:
uuid: "96b31954f32e42b690eed54de608887f"
format_stamp: "Formatted at 2026-06-20 14:30:36 on dist-test-slave-hfc3"
server_key: "67ce75a1385fd0289dca91b22591502e"
server_key_iv: "8096abd011c819ffba7a51a152642eb9"
server_key_version: "encryptionkey@0"
I20260620 14:30:36.961433 22627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/wal/instance:
uuid: "96b31954f32e42b690eed54de608887f"
format_stamp: "Formatted at 2026-06-20 14:30:36 on dist-test-slave-hfc3"
server_key: "67ce75a1385fd0289dca91b22591502e"
server_key_iv: "8096abd011c819ffba7a51a152642eb9"
server_key_version: "encryptionkey@0"
I20260620 14:30:36.962860 22627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:30:36.963593 22642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:36.963867 22627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:30:36.963970 22627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/wal
uuid: "96b31954f32e42b690eed54de608887f"
format_stamp: "Formatted at 2026-06-20 14:30:36 on dist-test-slave-hfc3"
server_key: "67ce75a1385fd0289dca91b22591502e"
server_key_iv: "8096abd011c819ffba7a51a152642eb9"
server_key_version: "encryptionkey@0"
I20260620 14:30:36.964044 22627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:36.964589 22620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } attempt: 2
I20260620 14:30:36.964991 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.964864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34674)
0620 14:30:36.964924 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.964962 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:30:36.965348 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.965254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34682)
0620 14:30:36.965306 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.965331 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:30:36.965447 22620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37379: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37379: connect: Connection refused (error 111)
I20260620 14:30:36.981043 22627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:36.983435 22627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:36.983603 22627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:36.987654 22627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37379
I20260620 14:30:36.987703 22694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37379 every 8 connection(s)
I20260620 14:30:36.988059 22627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:30:36.989012 22695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:36.990700 22695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:36.990883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLb3bhr/build/release/bin/kudu as pid 22627
I20260620 14:30:36.991003 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLb3bhr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965454096380-392-0/minicluster-data/master-2/wal/instance
I20260620 14:30:36.991202 392 external_mini_cluster.cc:1428] Setting key 4de45f8b1275fa02b7e0bb980fbb7a04
Jun 20 14:30:36 dist-test-slave-hfc3 krb5kdc[22464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:30:36 dist-test-slave-hfc3 krb5kdc[22464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:30:37.000313 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.991436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.0.98.62:46095)
0620 14:30:36.991513 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:36.991516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:36.991782 (+ 266us) server_negotiation.cc:407] Connection header received
0620 14:30:36.992351 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:36.992353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:36.992366 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:36.992417 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:36.992963 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.993433 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.994380 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.994579 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.994662 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.997035 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:36.997036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:36.997036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:36.997044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:36.998976 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.999383 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.999384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.999384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.999444 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.999685 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.999686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.999686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.999809 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:36.999887 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:37.000142 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 14:30:37.000220 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:30:37.000519 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.991445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46095 (local address 127.0.0.1:60736)
0620 14:30:36.991636 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:36.991681 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:36.992243 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:36.992445 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:36.992447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:36.992710 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:36.992865 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:36.992869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.993520 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.993521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:36.994254 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:36.994256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.994693 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.994694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:36.994803 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.995320 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:36.995329 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:36.996947 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:36.999050 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:36.999057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:36.999058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:36.999310 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:36.999514 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:36.999514 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:36.999515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:36.999602 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:36.999878 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:36.999884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:37.000012 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 14:30:37.000119 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:30:37.000263 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":41,"thread_start_us":103,"threads_started":1}
I20260620 14:30:37.000684 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.991758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.0.98.60:37379)
0620 14:30:36.991890 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:36.991895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:36.991925 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:30:36.992256 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:36.992267 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:36.992290 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:36.992428 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:36.993121 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.993925 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.994719 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.995006 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.995080 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.997637 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:36.997644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:36.997645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:36.997652 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:36.999549 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.999858 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.999861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.999862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.999917 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.000116 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.000117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.000117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.000255 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:37.000410 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:37.000490 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:30:37.000565 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260620 14:30:37.001241 22695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:37.002278 22545 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } attempt: 2
Jun 20 14:30:36 dist-test-slave-hfc3 krb5kdc[22464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:30:36 dist-test-slave-hfc3 krb5kdc[22464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:30:37 dist-test-slave-hfc3 krb5kdc[22464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:30:37.010037 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.001585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.0.98.61:43853)
0620 14:30:37.001631 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.001633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.001663 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:30:37.001738 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.001739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.001749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.001798 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.002073 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.002698 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.003551 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.003767 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.003848 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.006122 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:37.006123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:37.006124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:37.006132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:37.008664 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.009083 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.009084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.009084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.009138 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.009398 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.009399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.009399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.009552 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:37.009596 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:37.009822 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 14:30:37.009912 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:30:37.010044 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.001530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.0.98.62:46095)
0620 14:30:37.001569 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.001570 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.001579 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:30:37.001633 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.001633 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.001643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.001685 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.002137 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.002927 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.003576 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.003843 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.003902 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.006564 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:37.006565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:37.006565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:37.006572 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:37.008920 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.009275 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.009276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.009277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.009329 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.009526 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.009527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.009528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.009637 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:37.009822 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:37.009860 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 14:30:37.009936 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:30:37.010414 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.001492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43853 (local address 127.0.0.1:56728)
0620 14:30:37.001622 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:37.001632 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:37.001678 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:37.001798 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:37.001798 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:37.001938 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:37.001986 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:37.001988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.003014 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.003015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:37.003466 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:37.003467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.003854 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.003855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:37.003925 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.004366 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:37.004370 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:37.006005 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:37.008761 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:37.008762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:37.008763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:37.009018 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:37.009207 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:37.009208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:37.009208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:37.009272 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:37.009649 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:37.009650 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:37.009697 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:30:37.010169 (+ 472us) client_negotiation.cc:240] Negotiation successful
0620 14:30:37.010269 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 14:30:37.010571 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.002541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34704)
0620 14:30:37.002573 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:37.002583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:37.002623 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:37.002771 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:37.002773 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:37.002912 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:37.002946 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:37.002947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.004308 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.004309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:37.004894 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:37.004898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.005407 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.005408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:37.005469 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.005876 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:37.005882 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:37.007458 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:37.009486 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:37.009486 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:37.009487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:37.009692 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:37.009912 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:37.009913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:37.009914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:37.009971 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:37.010246 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:37.010247 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:37.010290 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:30:37.010369 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:30:37.010459 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 14:30:37.010787 22695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:37.010901 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.002579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.0.98.60:37379)
0620 14:30:37.002618 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.002620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.002629 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:30:37.002689 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.002689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.002699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.002749 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.003600 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.004236 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.004973 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.005321 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.005392 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.007541 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:37.007542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:37.007543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:37.007549 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:37.009399 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.009772 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.009773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.009773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.009822 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.010042 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.010043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.010043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.010166 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:37.010668 (+ 502us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:37.010711 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:30:37.010785 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:30:37.012567 22545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b: Bootstrap starting.
I20260620 14:30:37.013339 22545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:37.013660 22545 log.cc:824] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:37.014520 22545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b: No bootstrap required, opened a new log
Jun 20 14:30:37 dist-test-slave-hfc3 krb5kdc[22464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:30:37 dist-test-slave-hfc3 krb5kdc[22464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:30:37.016687 22545 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } }
I20260620 14:30:37.016857 22545 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6fb647bd6340d4a3f3ad0dd7cefe7b, State: Initialized, Role: FOLLOWER
I20260620 14:30:37.016954 22545 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } }
I20260620 14:30:37.017357 22545 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:37.017279 22701 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } } }
I20260620 14:30:37.017618 22701 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:37.018744 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.011095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.0.98.60:37379)
0620 14:30:37.011124 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.011126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.011134 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:30:37.011260 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.011261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.011269 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.011312 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.011598 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.012037 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.012678 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.012903 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.012987 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.015080 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:37.015081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:37.015082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:37.015089 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:37.016700 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.017076 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.017077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.017078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.017132 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.017308 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.017309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.017309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.017481 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:37.017563 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:37.018463 (+ 900us) server_negotiation.cc:299] Negotiation successful
0620 14:30:37.018607 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":15}
I20260620 14:30:37.018671 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.011035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34708)
0620 14:30:37.011100 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:37.011109 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:37.011147 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:37.011343 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:37.011344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:37.011490 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:37.011515 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:37.011517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.012109 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.012110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:37.012586 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:37.012587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.012987 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.012988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:37.013036 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.013453 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:37.013457 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:37.014985 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:37.016788 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:37.016789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:37.016789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:37.016997 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:37.017196 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:37.017197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:37.017197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:37.017249 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:37.018274 (+ 1025us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:37.018275 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:37.018323 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:30:37.018425 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:30:37.018533 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:30:37.019220 22712 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:37.019277 22712 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:37.019855 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.011128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.0.98.61:43853)
0620 14:30:37.011165 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.011167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.011248 (+ 81us) server_negotiation.cc:407] Connection header received
0620 14:30:37.011308 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.011308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.011318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.011380 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.011971 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.012616 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.013569 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.013879 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.013951 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.016548 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:37.016549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:37.016550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:37.016557 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:37.018159 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.018710 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.018711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.018712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.018774 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.019030 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.019031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.019031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.019160 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:37.019652 (+ 492us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:37.019696 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 14:30:37.019774 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:30:37.020682 22695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f: Bootstrap starting.
I20260620 14:30:37.021076 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:37.021701 22695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:37.022022 22695 log.cc:824] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:37.022042 22519 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60750
I20260620 14:30:37.022181 22519 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:30:37.022231 22519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:37.022976 22695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f: No bootstrap required, opened a new log
I20260620 14:30:37.020030 22477 master_runner.cc:186] Error getting master registration for 127.0.98.61:43853: OK, instance_id { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" instance_seqno: 1781965836820701 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:30:37.023126 22519 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } }
I20260620 14:30:37.023222 22477 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:30:37.023393 22519 leader_election.cc:288] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [CANDIDATE]: Term 1 election: Requested vote from peers f2baf75ba96f4fa7881edc8f1feb0b56 (127.0.98.61:43853), 96b31954f32e42b690eed54de608887f (127.0.98.60:37379)
I20260620 14:30:37.025040 22695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } }
I20260620 14:30:37.025202 22695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b31954f32e42b690eed54de608887f, State: Initialized, Role: FOLLOWER
I20260620 14:30:37.025296 22695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } }
I20260620 14:30:37.025636 22695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:37.025691 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "96b31954f32e42b690eed54de608887f"
I20260620 14:30:37.025841 22669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:37.026104 22722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } } }
I20260620 14:30:37.026206 22722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:37.027678 22669 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd6fb647bd6340d4a3f3ad0dd7cefe7b in term 1.
I20260620 14:30:37.027923 22494 leader_election.cc:302] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b31954f32e42b690eed54de608887f, cd6fb647bd6340d4a3f3ad0dd7cefe7b; no voters:
I20260620 14:30:37.028064 22701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:37.028218 22701 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [term 1 LEADER]: Becoming Leader. State: Replica: cd6fb647bd6340d4a3f3ad0dd7cefe7b, State: Running, Role: LEADER
I20260620 14:30:37.028303 22701 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } }
I20260620 14:30:37.028803 22701 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd6fb647bd6340d4a3f3ad0dd7cefe7b. Latest consensus state: current_term: 1 leader_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } } }
I20260620 14:30:37.028899 22701 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:37.029474 22734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:37.029884 22734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 14:30:37.031481 22736 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:37.031545 22736 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:37.031083 22627 master_runner.cc:186] Error getting master registration for 127.0.98.61:43853: OK, instance_id { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" instance_seqno: 1781965836820701 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:30:37.034008 22627 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:30:37.037123 22669 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [term 1 FOLLOWER]: Refusing update from remote peer cd6fb647bd6340d4a3f3ad0dd7cefe7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:37.037566 22701 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:37.037844 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.024577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.0.98.60:37379)
0620 14:30:37.024624 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.024626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.025041 (+ 415us) server_negotiation.cc:407] Connection header received
0620 14:30:37.025199 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.025200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.025224 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.025258 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.026117 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.026873 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.027954 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.028238 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.028284 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.034555 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:37.034557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:37.034557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:37.034565 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:37.036307 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.036741 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.036742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.036742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.036804 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.037014 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.037015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.037015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.037143 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:37.037262 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:37.037677 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 14:30:37.037758 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:30:37.039247 22722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd6fb647bd6340d4a3f3ad0dd7cefe7b. Latest consensus state: current_term: 1 leader_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } } }
I20260620 14:30:37.039328 22722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:37.040014 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.024612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.0.98.62:46095)
0620 14:30:37.024690 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.024692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.030569 (+ 5877us) server_negotiation.cc:407] Connection header received
0620 14:30:37.030634 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.030635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.030658 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.030711 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.031876 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.032626 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.034250 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.034583 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.034623 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.036941 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:37.036943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:37.036944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:37.036951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:37.038726 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.039114 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.039115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.039115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.039169 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.039349 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.039350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.039350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.039471 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:37.039556 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:37.039791 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 14:30:37.039890 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:30:37.042024 22728 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } } }
I20260620 14:30:37.042135 22728 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:37.042716 22722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } } }
I20260620 14:30:37.042794 22722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:37.043233 22738 mvcc.cc:205] Tried to move back new op lower bound from 7298932068500684800 to 7298932068468912128. Current Snapshot: MvccSnapshot[applied={T|T < 7298932068500684800}]
I20260620 14:30:37.043520 22739 mvcc.cc:205] Tried to move back new op lower bound from 7298932068500684800 to 7298932068468912128. Current Snapshot: MvccSnapshot[applied={T|T < 7298932068500684800}]
I20260620 14:30:37.046095 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.025103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.0.98.61:43853)
0620 14:30:37.025148 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.025149 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.030719 (+ 5570us) server_negotiation.cc:407] Connection header received
0620 14:30:37.030785 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.030786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.030799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.030854 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.038631 (+ 7777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.039411 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.041252 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.041555 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.041618 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.043144 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:37.043145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:37.043146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:37.043153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:37.044804 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.045216 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.045218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.045218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.045276 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.045485 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.045486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.045487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.045626 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:37.045730 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:37.045830 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:30:37.045919 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:30:37.054891 22734 catalog_manager.cc:1347] Generated new cluster ID: 50c3247ffd074023b3039a2aadcadc5d
I20260620 14:30:37.054980 22734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:37.056573 22620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } attempt: 3
Jun 20 14:30:37 dist-test-slave-hfc3 krb5kdc[22464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:30:37.065693 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.056873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37379 (local address 127.0.0.1:34718)
0620 14:30:37.056972 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:37.056983 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:37.057023 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:37.057267 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:37.057268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:37.057459 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:37.057507 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:37.057509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.058603 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.058604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:37.059421 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:37.059423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.060431 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.060432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:37.060511 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.061005 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:37.061024 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:37.062603 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:37.064403 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:37.064404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:37.064405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:37.064627 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:37.064912 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:37.064913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:37.064913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:37.064997 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:37.065350 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:37.065351 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:37.065412 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 14:30:37.065476 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:30:37.065585 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 14:30:37.066242 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:37.057096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.0.98.60:37379)
0620 14:30:37.057138 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.057140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.057147 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:30:37.057170 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.057171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.057180 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.057226 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.057588 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.058507 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.059533 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:37.059847 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:37.059907 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:37.062694 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:37.062695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:37.062695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:37.062703 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:37.064325 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.064715 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.064717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.064717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.064805 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:37.065100 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:37.065101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:37.065101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:37.065248 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:37.065307 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:37.066071 (+ 764us) server_negotiation.cc:299] Negotiation successful
0620 14:30:37.066152 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:30:37.068241 22620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56: Bootstrap starting.
I20260620 14:30:37.068995 22620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:37.069286 22620 log.cc:824] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:37.071280 22620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56: No bootstrap required, opened a new log
I20260620 14:30:37.072062 22734 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:37.072799 22734 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:37.073312 22620 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } }
I20260620 14:30:37.073454 22620 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2baf75ba96f4fa7881edc8f1feb0b56, State: Initialized, Role: FOLLOWER
I20260620 14:30:37.073534 22620 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } }
I20260620 14:30:37.073951 22594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56"
I20260620 14:30:37.074008 22593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:37.075103 22593 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [term 1 FOLLOWER]: Refusing update from remote peer cd6fb647bd6340d4a3f3ad0dd7cefe7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:37.075646 22728 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:37.076550 22734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b: Generated new TSK 0
I20260620 14:30:37.076704 22594 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd6fb647bd6340d4a3f3ad0dd7cefe7b in term 1.
I20260620 14:30:37.076732 22734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:37.073892 22620 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:30:37.080850 22759 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:37.080974 22759 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:37.073875 22744 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } } }
I20260620 14:30:37.081405 22744 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:37.081476 22744 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } } }
I20260620 14:30:37.081542 22744 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:37.083165 22745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd6fb647bd6340d4a3f3ad0dd7cefe7b. Latest consensus state: current_term: 1 leader_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } } }
I20260620 14:30:37.083250 22745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:37.083659 22746 mvcc.cc:205] Tried to move back new op lower bound from 7298932068641525760 to 7298932068468912128. Current Snapshot: MvccSnapshot[applied={T|T < 7298932068500684800}]
I20260620 14:30:37.091450 22701 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6fb647bd6340d4a3f3ad0dd7cefe7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46095 } } peers { permanent_uuid: "f2baf75ba96f4fa7881edc8f1feb0b56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43853 } } peers { permanent_uuid: "96b31954f32e42b690eed54de608887f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37379 } } }
I20260620 14:30:37.091574 22701 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6fb647bd6340d4a3f3ad0dd7cefe7b [sys.catalog]: This master's current role is: LEADER
W20260620 14:30:37.108354 22699 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.106705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.0.98.60:37379)
0620 14:30:37.106752 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.106754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.107672 (+ 918us) server_negotiation.cc:407] Connection header received
0620 14:30:37.107779 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.107780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.107792 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.107824 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.108282 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34732: BlockingRecv error: recv got EOF from 127.0.0.1:34732 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:30:37.109110 22547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.106944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.0.98.62:46095)
0620 14:30:37.106983 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.106984 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.108618 (+ 1634us) server_negotiation.cc:407] Connection header received
0620 14:30:37.108714 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.108715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.108726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.108757 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.109051 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60764: BlockingRecv error: recv got EOF from 127.0.0.1:60764 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:30:37.109835 22624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.107399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.0.98.61:43853)
0620 14:30:37.107436 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.107438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.109324 (+ 1886us) server_negotiation.cc:407] Connection header received
0620 14:30:37.109422 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.109423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.109433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.109461 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.109760 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56750: BlockingRecv error: recv got EOF from 127.0.0.1:56750 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:30:37.115412 22624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.114637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.0.98.61:43853)
0620 14:30:37.114674 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.114676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.114877 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:30:37.114983 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.114984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.115002 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.115030 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.115350 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56756: BlockingRecv error: recv got EOF from 127.0.0.1:56756 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:30:37.118637 22624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.117701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.0.98.61:43853)
0620 14:30:37.117733 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.117735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.118019 (+ 284us) server_negotiation.cc:407] Connection header received
0620 14:30:37.118121 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.118122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.118134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.118163 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.118592 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56770: BlockingRecv error: recv got EOF from 127.0.0.1:56770 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 14:30:37.122427 22624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.121678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.0.98.61:43853)
0620 14:30:37.121711 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.121712 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.121934 (+ 222us) server_negotiation.cc:407] Connection header received
0620 14:30:37.122023 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.122024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.122033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.122063 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.122387 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56776: BlockingRecv error: recv got EOF from 127.0.0.1:56776 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:30:37.128474 22624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.127507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.0.98.61:43853)
0620 14:30:37.127646 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.127648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.127923 (+ 275us) server_negotiation.cc:407] Connection header received
0620 14:30:37.128028 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.128029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.128039 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.128069 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.128397 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56782: BlockingRecv error: recv got EOF from 127.0.0.1:56782 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:30:37.138681 22624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.137714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.0.98.61:43853)
0620 14:30:37.137757 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.137760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.138030 (+ 270us) server_negotiation.cc:407] Connection header received
0620 14:30:37.138135 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.138136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.138147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.138177 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.138614 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56788: BlockingRecv error: recv got EOF from 127.0.0.1:56788 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:30:37.157042 22624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.156029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.0.98.61:43853)
0620 14:30:37.156075 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.156077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.156384 (+ 307us) server_negotiation.cc:407] Connection header received
0620 14:30:37.156517 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.156518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.156530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.156564 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.156975 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56804: BlockingRecv error: recv got EOF from 127.0.0.1:56804 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 14:30:37.191504 22624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.190466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.0.98.61:43853)
0620 14:30:37.190526 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.190528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.190881 (+ 353us) server_negotiation.cc:407] Connection header received
0620 14:30:37.191009 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.191010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.191022 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.191053 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.191435 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56812: BlockingRecv error: recv got EOF from 127.0.0.1:56812 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 14:30:37.257855 22624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.257183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.0.98.61:43853)
0620 14:30:37.257277 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.257279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.257349 (+ 70us) server_negotiation.cc:407] Connection header received
0620 14:30:37.257394 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.257395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.257405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.257470 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.257785 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56828: BlockingRecv error: recv got EOF from 127.0.0.1:56828 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 14:30:37.388427 22624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.387577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.0.98.61:43853)
0620 14:30:37.387670 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.387672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.387761 (+ 89us) server_negotiation.cc:407] Connection header received
0620 14:30:37.387944 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.387945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.387958 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.388013 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.388361 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56838: BlockingRecv error: recv got EOF from 127.0.0.1:56838 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 14:30:37.646557 22624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:37.645691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.0.98.61:43853)
0620 14:30:37.645763 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:37.645765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:37.645834 (+ 69us) server_negotiation.cc:407] Connection header received
0620 14:30:37.645995 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:37.645996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:37.646010 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:37.646089 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:37.646488 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56852: BlockingRecv error: recv got EOF from 127.0.0.1:56852 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:30:38.032079 22736 catalog_manager.cc:1259] Loaded cluster ID: 50c3247ffd074023b3039a2aadcadc5d
I20260620 14:30:38.032163 22736 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f: loading cluster ID for follower catalog manager: success
I20260620 14:30:38.033028 22736 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f: acquiring CA information for follower catalog manager: success
I20260620 14:30:38.033366 22736 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96b31954f32e42b690eed54de608887f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:30:38.081456 22759 catalog_manager.cc:1259] Loaded cluster ID: 50c3247ffd074023b3039a2aadcadc5d
I20260620 14:30:38.081535 22759 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56: loading cluster ID for follower catalog manager: success
I20260620 14:30:38.082434 22759 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56: acquiring CA information for follower catalog manager: success
I20260620 14:30:38.082757 22759 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2baf75ba96f4fa7881edc8f1feb0b56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:30:38.162896 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:38.159729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.0.98.61:43853)
0620 14:30:38.159878 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:38.159880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:38.159980 (+ 100us) server_negotiation.cc:407] Connection header received
0620 14:30:38.160081 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:38.160082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:38.160155 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:38.160252 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:38.160430 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:38.160960 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:38.161771 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:38.162059 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:38.162135 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:38.162426 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:38.162582 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:38.162676 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:38.162750 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:30:38.162778 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
I20260620 14:30:38.168159 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:38.165086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.0.98.60:37379)
0620 14:30:38.165620 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:38.165622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:38.165632 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:30:38.165681 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:38.165682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:38.165737 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:38.165854 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:38.165971 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:38.166517 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:38.167294 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:38.167489 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:38.167569 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:38.167777 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:38.167882 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:38.167971 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:38.168025 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:30:38.168048 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":54,"threads_started":1}
I20260620 14:30:38.173190 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:38.170007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.0.98.62:46095)
0620 14:30:38.170295 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:38.170297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:38.170304 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:30:38.170421 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:38.170422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:38.170496 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:38.170573 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:38.170698 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:38.171426 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:38.172177 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:38.172454 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:38.172518 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:38.172718 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:38.172843 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:38.172904 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:38.173079 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:30:38.173099 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":63,"threads_started":1}
I20260620 14:30:38.173384 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:38.170180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.0.98.61:43853)
0620 14:30:38.170234 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:38.170236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:38.170336 (+ 100us) server_negotiation.cc:407] Connection header received
0620 14:30:38.170457 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:38.170458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:38.170471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:38.170502 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:38.170722 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:38.171197 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:38.172393 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:38.172687 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:38.172755 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:38.172910 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:38.172944 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:38.173001 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:38.173280 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:30:38.173309 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:30:38.173633 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:38.170123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.0.98.60:37379)
0620 14:30:38.170235 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:38.170237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:38.170246 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:30:38.170328 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:38.170349 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:38.170362 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:38.170415 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:38.170522 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:38.171281 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:38.172363 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:38.172624 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:38.172701 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:38.172898 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:38.172930 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:38.172993 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:38.173190 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 14:30:38.173214 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 14:30:38.178551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 22477
I20260620 14:30:38.184285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 22552
I20260620 14:30:38.189509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLb3bhr/build/release/bin/kudu with pid 22627
2026-06-20T14:30:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2811 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5606 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (384099 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
2 FAILED TESTS